iCreatAI News
Skip to content
AWS Machine Learning Blog· Apoorva Chandra·· 2 days agoAI score40

How Salesforce is advancing global-scale model serving with cross-Region inference routing in Amazon SageMaker AI

How Salesforce is advancing global-scale model serving with cross-Region inference routing in Amazon SageMaker AI

AI summary

Salesforce's Agentforce Foundation team serves in-house and open source models across 30+ Regions and hundreds of AWS accounts, where per-Region hosting fragmented capacity and made traffic spikes an availability problem. Using SageMaker AI's inference router, Salesforce replaced its Internal Gateway's self-built health checks, retries and Region-switching logic with a managed routing layer that prefers the local Region, shifts traffic on 5xx errors, throttles and timeouts, and keeps data-residency-tagged requests pinned in-Region. In a failover test, traffic began moving to a remote Region about two minutes after the first errors, peaked at roughly 78 percent, and returned gradually over about 10 minutes, with cross-Region latency adding under 100 milliseconds at p50.

Source: AWS Machine Learning Blog · aws.amazon.com