Stanford Re-Test: 44% of LLM Findings No Longer Hold

1. 03:41
   The Decoder

   Rapid LLM development is outrunning the research meant to explain it, study finds

   Stanford researchers Owen Queen and James Zou tested 50 widely cited claims about large language models against 20 models and found 44 percent have already "expired," with an average lifespan of 15.7 months among the 22 findings with an estimated expiration date. Claims about what models can't do expired fastest, while findings on preferences, sycophancy and evaluator behavior proved longer-lived, and results varied widely across models even within the same release window. The researchers argue empirical LLM findings should be presented as observations about specific models at a specific point, with model, version, date, prompts and analysis code documented for re-testing.

Trends compare the same fully observed participants, which may cover fewer participants than current activity. Move the pointer or tap the chart to inspect hourly activity; use the left and right arrow keys with a keyboard.