Building LLM-Driven Workflows — Large Language Models (LLMs) continue to take the world by storm. Hosting these models is a challenging task as we’ve explored in my previous articles. The next challenge is operationalizing these hosted LLMs in larger real-world applications. To solve these two problems we have a pair of respective tools that…

Integrate Triton Inference Server With Amazon SageMaker — In the past we’ve explored SageMaker Multi-Model Endpoints (MME) as a cost effective option to host multiple models behind a singular endpoint. While hosting smaller models is possible on MME with CPU based instances, as these models get larger and more complex in nature sometimes GPU compute may be necessary. …

A flexible high-performant model serving solution — Machine Learning’s (ML) value is truly recognized in real-world applications when we arrive at Model Hosting and Inference. It’s hard to productionize ML workloads if you don’t have a highly performant model-serving solution that will help your model scale up and down. What is a model server/what is model serving…

Pick the option that makes the most sense for your use-case — Amazon SageMaker is a platform that can be utilized for advanced Machine Learning Model Hosting. As Generative AI continues to expand at a rapid rate, so do the challenges of hosting these large models. Many factors make hosting these models particularly difficult ranging from model size to being able to…

Another way to efficiently host and scale your LLMs with Amazon SageMaker — Large Language Models (LLMs) continue to soar in popularity as a new one is released nearly every week. With the number of these models increasing, so are the options for how we can host them. In my previous article we explored how we could utilize DJL Serving within Amazon SageMaker…

An Alternative To SageMaker Local Mode — A pain point with getting started with SageMaker Real-Time Inference is that it is hard to debug at times. When creating an endpoint there are a number of ingredients you need to make sure are baked properly for successful deployment. Proper file structuring of model artifacts depending on the Model…

Deploy BART on Amazon SageMaker Real-Time Inference — Large Language Models (LLMs) and Generative AI continue to take over the Machine Learning and general tech space in 2023. With the LLM expansion has come an influx of new models that continue to improve at a stunning rate. While the accuracy and performance of these models are incredible, they…

Scale and Host LLMs on AWS — Large Language Models (LLMs) and Generative AI are accelerating Machine Learning growth across various industries. With LLMs the scope for Machine Learning has increased to incredible heights, but has also been accompanied with a new set of challenges. The size of LLMs lead to difficult problems in both the Training…

Harness SageMaker Pipelines With Batch Inference — MLOPs continues to be a popular topic in the Machine Learning world and for good reason. As ML continues to mature it’s become crucial that we apply Software Engineering best practices to make reusable, end to end workflows tailored for Data Science. A tool we explored in my previous article…