AWS Machine Learning Blog· Dmitry Soldatkin·· 1 days agoAI score58
Introducing Amazon SageMaker HyperPod Global Inference Gateway: One endpoint across Regions and clusters
Introducing Amazon SageMaker HyperPod Global Inference Gateway: One endpoint across Regions and clusters
AI summary
Amazon SageMaker HyperPod introduces the Global Inference Gateway, a SageMaker-managed endpoint that routes requests across every registered HyperPod cluster in one Region or across Regions, with automatic failover and SigV4/IAM authentication, while each cluster's existing Inference Gateway still selects the pod. The gateway scores healthy clusters on KV cache headroom, queue depth, Region proximity, and error rate, with configurable weights, and carries traffic over AWS PrivateLink. It is available in 13 AWS Regions.
Source: AWS Machine Learning Blog · aws.amazon.com