iCreatAI News
Skip to content
AWS Machine Learning Blog· Dmitry Soldatkin·· 4 hr agoAI score58

Amazon SageMaker AI real-time endpoints now support KV cache aware routing

Amazon SageMaker AI real-time endpoints now support KV cache aware routing

AI summary

Amazon SageMaker AI introduced KV cache-aware routing for real-time endpoints, which uses cache state reported by the model container to send each request to an instance holding a long cached match for its prompt while weighing instance load. It is enabled by setting SM_VLLM_KV_EVENTS_CONFIG in the model environment and RoutingStrategy to KV_AWARE in the endpoint configuration, and requires at least two instances or inference component copies. In benchmarks with Llama 3.1 70B Instruct on seven ml.p5.48xlarge instances, the company reports p50 time to first token reduced by up to 97 percent and throughput increased by up to 234 percent versus random routing.

Source: AWS Machine Learning Blog · aws.amazon.com