Rapid LLM development is outrunning the research meant to explain it, study finds

Stanford researchers Owen Queen and James Zou tested 50 widely cited claims about large language models against 20 models and found 44 percent have already "expired," with an average lifespan of 15.7 months among the 22 findings with an estimated expiration date. Claims about what models can't do expired fastest, while findings on preferences, sycophancy and evaluator behavior proved longer-lived, and results varied widely across models even within the same release window. The researchers argue empirical LLM findings should be presented as observations about specific models at a specific point, with model, version, date, prompts and analysis code documented for re-testing.