iCreatAI News
Skip to content
Trending storyUpdating

SageMaker AI Real-Time Endpoints Add KV Cache-Aware Routing

1 reports1 sources4 hr agoUpdated

Understand this story

AI overview

Amazon SageMaker AI introduced a KV cache-aware routing strategy (KV_AWARE) for real-time endpoints: it uses cache status reported by model containers to send requests to the instance with the longest cache match that is not overloaded. AWS said the feature comes at no additional charge. The feature is available in all commercial AWS Regions that support SageMaker AI real-time inference, but only for endpoints using supported containers. At launch it supports the AWS vLLM Deep Learning Container (vLLM 0.33.0 or later images, or Amazon Linux builds server-sagemaker-cuda-v2.8.0 or later). Routing only takes effect when a single model endpoint has at least two instances, or an inference component has at least two copies, which may reside on the same instance.

AI overview based on source reports · Updated 3 hr ago

Coverage timeline

Follow the coverage to explore different perspectives.

Nov 18
  1. AWS Machine Learning Blog
    Amazon SageMaker AI real-time endpoints now support KV cache aware routing

    Amazon SageMaker AI introduced KV cache-aware routing for real-time endpoints, which uses cache state reported by the model container to send each request to an instance holding a long cached match for its prompt while weighing instance load. It is enabled by setting SM_VLLM_KV_EVENTS_CONFIG in the model environment and RoutingStrategy to KV_AWARE in the endpoint configuration, and requires at least two instances or inference component copies. In benchmarks with Llama 3.1 70B Instruct on seven ml.p5.48xlarge instances, the company reports p50 time to first token reduced by up to 97 percent and throughput increased by up to 234 percent versus random routing.

Activity for this story

Current comparable activity
9
Peak comparable activity
10Nov 18 06:00
Change in the past 24 hours
–

Trends compare the same fully observed participants, which may cover fewer participants than current activity. Move the pointer or tap the chart to inspect hourly activity; use the left and right arrow keys with a keyboard.