iCreatAI News
Skip to content
AWS Machine Learning Blog· Summer Devlin·· 2 days agoAI score34

Exa boosts embedding throughput by 50% to index billions of documents on AWS AI chips

Exa boosts embedding throughput by 50% to index billions of documents on AWS AI chips

AI summary

Exa describes optimizing its embedding infrastructure on AWS Neuron hardware (Inferentia2 and Trainium2) to cut per-document costs at the scale of tens or hundreds of billions of documents, reaching a 47% improvement in performance and cost per 1 million documents embedded versus its initial baseline. The gains came from weight layout optimization, autocast, NKI kernels including a custom fused SwiGLU MLP kernel, and shape-specific attention rewrites, with agents developing the optimizations in a Flyte-based harness that checks correctness at 0.999 cosine similarity. Exa says the Inferentia2 work carried over directly to Trainium2 and that scheduling across Regions and instance types by performance per dollar helps avoid compute shortages.

Source: AWS Machine Learning Blog · aws.amazon.com