StateBench: Testing how language models track state changes over long documents

Ai2 introduces StateBench, a long-context state-tracking benchmark with four difficulty levels (r-trivial, aperiodic, soluble, and general finite) grounded in Krohn–Rhodes theory. Testing open pretrained models and frontier models including Gemini 3.8 Flash, Claude Opus 5.5, and GPT-6 Sol and Luna, they found accuracy fell sharply on soluble and general finite problems as inputs grew longer, dropping to 13% or lower on the longest general finite problems. In from-scratch training of 275-million-parameter models, Gated DeltaNet and hybrid models generalized almost perfectly on the first three levels out to tens of thousands of assignments, while transformers never learned the hardest level.