Ai2 Releases StateBench, a Long-Context State Tracking Benchmark

1. 01:15
   Hugging Face Blog

   StateBench: Testing how language models track state changes over long documents

   Researchers introduce StateBench, a benchmark with four difficulty levels of long-context state tracking, from immutable assignment to general finite problems involving assignment by reference. In tests, open pretrained models and frontier models (Gemini 3.8 Flash, Claude Opus 5.5, GPT-6 Sol and Luna) degraded on harder levels as inputs grew, while 275M-parameter Gated DeltaNet and hybrid models trained directly on the tasks generalized far beyond training length, though transformers never learned the hardest level. The work is released with a tech report, data, and code, and notes open questions such as fixed variable counts and the exact hierarchy placement of constrained Gated DeltaNet variants.

1. 16:00
   Allen Institute for AI Blog

   StateBench: Testing how language models track state changes over long documents

   Ai2 introduces StateBench, a long-context state-tracking benchmark with four difficulty levels (r-trivial, aperiodic, soluble, and general finite) grounded in Krohn–Rhodes theory. Testing open pretrained models and frontier models including Gemini 3.8 Flash, Claude Opus 5.5, and GPT-6 Sol and Luna, they found accuracy fell sharply on soluble and general finite problems as inputs grew longer, dropping to 13% or lower on the longest general finite problems. In from-scratch training of 275-million-parameter models, Gated DeltaNet and hybrid models generalized almost perfectly on the first three levels out to tens of thousands of assignments, while transformers never learned the hardest level.

Trends compare the same fully observed participants, which may cover fewer participants than current activity. Move the pointer or tap the chart to inspect hourly activity; use the left and right arrow keys with a keyboard.