Exa boosts embedding throughput by 50% on AWS Neuron chips
Understand this story
Exa says it optimized its embedding inference infrastructure with AWS's Inferentia2 and Trainium2 chips (i.e., AWS Neuron hardware), increasing embedding throughput by about 1.5x (50%) to support web indexing at the scale of billions of documents, while lowering inference costs and improving flexibility. Exa says that after reducing numerical precision, the output achieves a cosine similarity of 0.999 compared with the reference results, meeting its correctness threshold, with the accuracy impact within an acceptable range. The optimization currently covers only Inferentia2 in the US West (Oregon) region, and Exa also notes that tuning for only one type of hardware in a single region has clear limitations.
AI overview based on source reports · Updated 2 days ago
Coverage timeline
Follow the coverage to explore different perspectives.
- AWS Machine Learning BlogExa boosts embedding throughput by 50% to index billions of documents on AWS AI chips
Exa describes optimizing its embedding infrastructure on AWS Neuron hardware (Inferentia2 and Trainium2) to cut per-document costs at the scale of tens or hundreds of billions of documents, reaching a 47% improvement in performance and cost per 1 million documents embedded versus its initial baseline. The gains came from weight layout optimization, autocast, NKI kernels including a custom fused SwiGLU MLP kernel, and shape-specific attention rewrites, with agents developing the optimizations in a Flyte-based harness that checks correctness at 0.999 cosine similarity. Exa says the Inferentia2 work carried over directly to Trainium2 and that scheduling across Regions and instance types by performance per dollar helps avoid compute shortages.
Activity for this story
- Current comparable activity
- 4
- Peak comparable activity
- 10Nov 19 09:00
- Change in the past 24 hours
- -51%
Trends compare the same fully observed participants, which may cover fewer participants than current activity. Move the pointer or tap the chart to inspect hourly activity; use the left and right arrow keys with a keyboard.