SageMaker Inference now supports Multi-Instance GPU to maximize GPU utilization for model endpoints

Amazon SageMaker Inference now supports NVIDIA Multi-Instance GPU (MIG) partitioning on inference component endpoints, letting you divide a physical GPU into up to seven hardware-isolated partitions and place a different model in each. Partitioning is declared via AcceleratorPartitionLayouts on the endpoint config and AcceleratorPartitionType on each inference component, with no new API operations or IAM actions. In testing, a mig-3g.40gb partition of an H100 delivered 53–89% more output tokens per second and 18–34% lower cost per million output tokens than a full A100-40GB above a workload-dependent concurrency crossover.