SageMaker Experiments

Organizing and Tracking your ML Training and Experimentation

Ram Vegiraju
AWS in Plain English
7 min readJan 6, 2023

--

Image from Unsplash by Louis Reed

MLOps is a growing and crucial domain in the field of Machine Learning. I’ve been in Data Science and Medium for a few years now and have never seen such an implosion of tools in this area ranging from MLFlow to SageMaker Pipelines.

A key part of MLOps is experimentation. Often times in the real-world Data Scientists will have numerous models, training sets, different hyperparameter combinations amongst many variables they will have to test before taking a model to production. Manually tracking all these different knobs can be difficult and it is a crucial part of the MLOps lifecycle to properly capture all this information.

Amazon SageMaker has recently upgraded SageMaker Experiments to make it simpler than ever to track different training runs, parameters, and metrics as you conduct your ML experiments. In this article today, we’ll take a look at a sample example of how you can utilize Experiments to organize and track your ML model trainings.

NOTE: This article will assume basic knowledge of AWS and SageMaker specific sub-features such as Studio (the SageMaker UI) and interacting with SageMaker and AWS as a whole via the SageMaker Python SDK and Boto3 AWS Python SDK. We will not focus on model building in this article, but we’ll be using the Sklearn framework to build a sample model.

Setup

For development we’ll be working in the SageMaker Studio console with a ml.t3.medium Data Science Kernel. This is a tailored UI for SageMaker and also comes with a host of pre-built images for popular frameworks and compute instance types backing the notebooks. You could run this notebook locally in your own environment, but the UI features of Studio help you visualize SageMaker Experiments so it’s strongly encouraged to work in this environment.

To work with SageMaker we need two main packages: SageMaker Python SDK and Boto3 AWS Python SDK. The SageMaker Python SDK contains the latest and simplest ways to interface with the SageMaker Experiments feature.

# update boto3 and sagemaker to ensure latest SDK version
import sys
!{sys.executable} -m pip install --upgrade pip
!{sys.executable} -m pip install --upgrade boto3…

--

--