SageMaker HyperPod Inference supports NVIDIA Dynamo
Understand this story
Amazon SageMaker HyperPod Inference has added support for the NVIDIA Dynamo distributed inference framework, combining Dynamo's multi-node deployment, prefill and decode separation, and KV-aware routing with HyperPod's EFA networking, GPU scheduling, and node health recovery. To adopt a supported Dynamo version, the add-on must be updated. Inference engine options include vLLM, SGLang, or TensorRT-LLM, with vLLM used by default.
AI overview based on source reports · Updated 1 hr ago
Coverage timeline
Follow the coverage to explore different perspectives.
- AWS Machine Learning BlogManaged NVIDIA Dynamo on Amazon SageMaker HyperPod Inference
Amazon SageMaker HyperPod Inference now supports NVIDIA Dynamo, an open source distributed inference framework offering multi-node deployment, disaggregated prefill and decode, and KV-aware routing with engines such as vLLM, SGLang, and TensorRT-LLM, backed by EFA networking, GPU scheduling, and node health monitoring. Deployments can use a bring-your-own DynamoGraphDeployment or an InferenceEndpointConfig with a dynamoSpec section, and benchmarks showed disaggregated serving reaching 2.05× the highest aggregated throughput for Qwen3-235B-A22B-FP8 (207 versus 101 output tokens per second per GPU) and a two-node configuration retaining 99.1 percent of single-node throughput for Qwen3.5-397B-A17B-FP8.
Activity for this story
Not enough continuous observations to show a trend yet.