NVIDIA Technical Blog· Timmy Liu·· 8 hr agoAI score34
Co-Designing Sparse Attention for Efficient Long-Context Inference
Co-Designing Sparse Attention for Efficient Long-Context Inference
AI summary
Agentic workloads, long documents, and large codebases are pushing context lengths from tens of thousands to hundreds of thousands of tokens, where attention becomes the most expensive part of a transformer layer. Dense attention scores every query against every key, so its cost per query grows with context length N, and at 128K tokens it can take most of the prefill time.
Source: NVIDIA Technical Blog · developer.nvidia.com