iCreatAI News
Skip to content
AWS Machine Learning Blog· Vinay Arora·· 2 hr agoAI score58

Managed NVIDIA Dynamo on Amazon SageMaker HyperPod Inference

Managed NVIDIA Dynamo on Amazon SageMaker HyperPod Inference

AI summary

Amazon SageMaker HyperPod Inference now supports NVIDIA Dynamo, an open source distributed inference framework offering multi-node deployment, disaggregated prefill and decode, and KV-aware routing with engines such as vLLM, SGLang, and TensorRT-LLM, backed by EFA networking, GPU scheduling, and node health monitoring. Deployments can use a bring-your-own DynamoGraphDeployment or an InferenceEndpointConfig with a dynamoSpec section, and benchmarks showed disaggregated serving reaching 2.05× the highest aggregated throughput for Qwen3-235B-A22B-FP8 (207 versus 101 output tokens per second per GPU) and a two-node configuration retaining 99.1 percent of single-node throughput for Qwen3.5-397B-A17B-FP8.

Source: AWS Machine Learning Blog · aws.amazon.com