iCreatAI News
Skip to content
Trending storyPast story

Zyphra releases hierarchical sparse attention method ZHA

1 reports1 sources7 days agoUpdated

Understand this story

AI overview

Zyphra releases the sparse attention method Zyphra Hierarchical Attention (ZHA), whose core is using a tree-structured indexer to make the index search itself sparse as well, moving toward subquadratic attention. Zyphra says the method can be distilled into a frozen pretrained model with only about 4B tokens, and during training only reference queries participate in updates while the base model remains frozen. An optimized kernel has not yet been released; the reported speedups come from roofline modeling based on H100 BF16 rather than actual measurements.

AI overview based on source reports · Updated 6 days ago

Coverage timeline

Follow the coverage to explore different perspectives.

Nov 3
  1. MarkTechPost
    Zyphra AI Introduces Zyphra Hierarchical Attention (ZHA): A Path to True Subquadratic Attention

    Zyphra released Zyphra Hierarchical Attention (ZHA), a sparse attention method that uses a tree-structured indexer to search the KV cache instead of scanning the full context, and its paper shows it can be distilled into frozen pretrained models with about 4B tokens. On Qwen3.6-35B-A3B, ZHA adds 81,920 trainable parameters and stays within a few points of dense attention on seven reasoning, coding and retrieval benchmarks, with the wide tree reaching 96.2% on RULER-13 at 256K versus 96.0% dense, while MSA fell to 77.1%. Zyphra released no optimized kernels, so speedups such as 373 to 417x over dense at 1B tokens come from an H100 BF16 roofline model rather than measurements.

Activity for this story

Current comparable activity
3
Peak comparable activity
9Nov 3 12:00
Change in the past 24 hours
-50%

Trends compare the same fully observed participants, which may cover fewer participants than current activity. Move the pointer or tap the chart to inspect hourly activity; use the left and right arrow keys with a keyboard.