Managed NVIDIA Dynamo on Amazon SageMaker HyperPod Inference
Managed NVIDIA Dynamo on Amazon SageMaker HyperPod Inference
Amazon SageMaker HyperPod Inference now supports NVIDIA Dynamo, an open source distributed inference framework offering multi-node deployment, disaggregated prefill and decode, and KV-aware routing with engines such as vLLM, SGLang, and TensorRT-LLM, backed by EFA networking, GPU scheduling, and node health monitoring. Deployments can use a bring-your-own DynamoGraphDeployment or an InferenceEndpointConfig with a dynamoSpec section, and benchmarks showed disaggregated serving reaching 2.05× the highest aggregated throughput for Qwen3-235B-A22B-FP8 (207 versus 101 output tokens per second per GPU) and a two-node configuration retaining 99.1 percent of single-node throughput for Qwen3.5-397B-A17B-FP8.
Source: AWS Machine Learning Blog · aws.amazon.com