iCreatAI News
Skip to content
Trending storyUpdating

SageMaker HyperPod Inference supports NVIDIA Dynamo

1 reports1 sources2 hr agoUpdated

Understand this story

AI overview

Amazon SageMaker HyperPod Inference has added support for the NVIDIA Dynamo distributed inference framework, combining Dynamo's multi-node deployment, prefill and decode separation, and KV-aware routing with HyperPod's EFA networking, GPU scheduling, and node health recovery. To adopt a supported Dynamo version, the add-on must be updated. Inference engine options include vLLM, SGLang, or TensorRT-LLM, with vLLM used by default.

AI overview based on source reports · Updated 1 hr ago

Coverage timeline

Follow the coverage to explore different perspectives.

Nov 18
  1. AWS Machine Learning Blog
    Managed NVIDIA Dynamo on Amazon SageMaker HyperPod Inference

    Amazon SageMaker HyperPod Inference now supports NVIDIA Dynamo, an open source distributed inference framework offering multi-node deployment, disaggregated prefill and decode, and KV-aware routing with engines such as vLLM, SGLang, and TensorRT-LLM, backed by EFA networking, GPU scheduling, and node health monitoring. Deployments can use a bring-your-own DynamoGraphDeployment or an InferenceEndpointConfig with a dynamoSpec section, and benchmarks showed disaggregated serving reaching 2.05× the highest aggregated throughput for Qwen3-235B-A22B-FP8 (207 versus 101 output tokens per second per GPU) and a two-node configuration retaining 99.1 percent of single-node throughput for Qwen3.5-397B-A17B-FP8.

Activity for this story

Not enough continuous observations to show a trend yet.