Andy Hall reproduces METR's AI auditing research
Understand this story
Anthropic Institute researcher Andy Hall reproduced and extended METR's research on AI auditing, testing whether GPT-5, GPT-5 nano, GPT-6 Astra, Opus 4.8 and Gemini 3.8 Flash can identify human-annotated agent violations at a 5% false positive rate. The so-called 5% false positive rate means that on average 5 out of every 100 clean logs are incorrectly flagged as suspicious. Hall said that while keeping METR's original prompts largely unchanged for comparability, Fable 5 and Opus 5 completely refused to perform the evaluation. The reward hacking data is based on only 47 cases, and a single case shifts the rate by about two percentage points; the sandbagging cases are mostly synthetic: METR researchers first instructed the models to sabotage, then removed those instructions from the visible chain-of-thought logs. The conclusion is: auditing models can catch most violations, but the volume of false positives would overwhelm human review.
AI overview based on source reports · Updated 14 min ago
Coverage timeline
Follow the coverage to explore different perspectives.
- The DecoderAI auditors catch most agent misbehavior but flood reviewers with false alarms, new study finds
In a Substack post, Anthropic Institute researcher Andy Hall replicated and extended METR's MALT study by testing GPT-5, GPT-5 nano, GPT-6 Astra, Opus 4.8 and Gemini 3.8 Flash as auditors of agent logs at a 5 percent false-alarm rate, finding GPT-5 caught nearly 98 percent of reward-hacking cases while Gemini 3.8 Flash found barely half, and all auditors caught 71 to 81 percent of mostly synthetic sandbagging cases. Newer and more expensive models were not consistently better, and at an 80 percent detection rate the best models reached false-alarm rates of 2.5 to 3 percent while Opus 4.8 and Gemini 3.8 Flash flagged 40 and 63 percent of harmless logs. Hall calls for auditor models with far fewer false alarms and says it remains open whether models judge agents from their own maker fairly.
Activity for this story
Not enough continuous observations to show a trend yet.