AWS Machine Learning Blog· Shubham Gupta·· 10 hr agoAI score38
How ArisGlobal scaled generative AI inference on Amazon Bedrock
How ArisGlobal scaled generative AI inference on Amazon Bedrock
AI summary
ArisGlobal's NavaX AI engine on Amazon Bedrock reversed its prompt order to cache the shared prompt prefix instead of the per-case document, after the shared prefix grew about 150% over six months and per-case inference costs rose more than 200%. According to ArisGlobal's production metrics, 65% of input tokens are now served from cache, cutting effective input cost by roughly 50% and overall monthly inference spend by about 40% at more than 19 billion tokens per month, while EU cross-Region inference routing stayed unchanged.
Source: AWS Machine Learning Blog · aws.amazon.com