Zyphra releases hierarchical sparse attention method ZHA
Understand this story
Zyphra releases the sparse attention method Zyphra Hierarchical Attention (ZHA), whose core is using a tree-structured indexer to make the index search itself sparse as well, moving toward subquadratic attention. Zyphra says the method can be distilled into a frozen pretrained model with only about 4B tokens, and during training only reference queries participate in updates while the base model remains frozen. An optimized kernel has not yet been released; the reported speedups come from roofline modeling based on H100 BF16 rather than actual measurements.
AI overview based on source reports · Updated 6 days ago
Coverage timeline
Follow the coverage to explore different perspectives.
- MarkTechPostZyphra AI Introduces Zyphra Hierarchical Attention (ZHA): A Path to True Subquadratic Attention
Zyphra released Zyphra Hierarchical Attention (ZHA), a sparse attention method that uses a tree-structured indexer to search the KV cache instead of scanning the full context, and its paper shows it can be distilled into frozen pretrained models with about 4B tokens. On Qwen3.6-35B-A3B, ZHA adds 81,920 trainable parameters and stays within a few points of dense attention on seven reasoning, coding and retrieval benchmarks, with the wide tree reaching 96.2% on RULER-13 at 256K versus 96.0% dense, while MSA fell to 77.1%. Zyphra released no optimized kernels, so speedups such as 373 to 417x over dense at 1B tokens come from an H100 BF16 roofline model rather than measurements.
Activity for this story
- Current comparable activity
- 3
- Peak comparable activity
- 9Nov 3 12:00
- Change in the past 24 hours
- -50%
Trends compare the same fully observed participants, which may cover fewer participants than current activity. Move the pointer or tap the chart to inspect hourly activity; use the left and right arrow keys with a keyboard.