SageMaker Inference adds MIG GPU partitioning capability
Understand this story
Amazon SageMaker Inference adds a GPU partitioning capability that, based on NVIDIA MIG, splits a single physical GPU into mutually isolated partitions, letting multiple models each use independent compute on the same GPU. This capability applies only to endpoints that use inference components, i.e., scenarios where multiple models are deployed to the same endpoint with compute set separately for each; single-model endpoints, serverless endpoints, and heterogeneous instance pools do not support partitioning. The available instance families are P4d, P5, G7, and G7e. AWS says billing is unchanged and still charged per whole-GPU instance.
AI overview based on source reports · Updated 2 hr ago
Coverage timeline
Follow the coverage to explore different perspectives.
- AWS Machine Learning BlogSageMaker Inference now supports Multi-Instance GPU to maximize GPU utilization for model endpoints
Amazon SageMaker Inference now supports NVIDIA Multi-Instance GPU (MIG) partitioning on inference component endpoints, letting you divide a physical GPU into up to seven hardware-isolated partitions and place a different model in each. Partitioning is declared via AcceleratorPartitionLayouts on the endpoint config and AcceleratorPartitionType on each inference component, with no new API operations or IAM actions. In testing, a mig-3g.40gb partition of an H100 delivered 53–89% more output tokens per second and 18–34% lower cost per million output tokens than a full A100-40GB above a workload-dependent concurrency crossover.
Activity for this story
Not enough continuous observations to show a trend yet.