Audit finds AgentDojo missed 433 agent attacks
Understand this story
TRUSCOR has released a preprint auditing the automatic determination of attack success by the agent security benchmark AgentDojo. Of 19,381 published runs, the benchmark and the audit agreed on 97.7% of non-crashing runs, but 433 runs in which the agent had executed key attacker actions were judged by the benchmark as having no attack, accounting for 11.9% of the runs the audit identified as attacks; among new runs in v1.2.2, 55 of 744 attacks were missed. TRUSCOR has also open-sourced the audit tool sova-oss 0.1.0 (PyPI package name pip install sova-oss), with text and data under CC BY 4.0 and code under the Apache-2.0 license. The paper has not yet undergone peer review.
AI overview based on source reports · Updated 19 hr ago
Coverage timeline
Follow the coverage to explore different perspectives.
- Hacker News AIDid the attack work? Auditing AI-agent security benchmark verdicts
The author audited AgentDojo's automated attack-success verdicts against agents' actual tool calls, finding 97.7% agreement on non-crashing published runs but 433 runs scored as no attack in which the agent executed the attacker's decisive action (11.9% of audited attacks), and 55 of 744 missed on new runs under benchmark v1.2.2. Re-running 240 recorded attacks ten times each, 224 recurred in at least half of runs but only 201 could be shown at 95% confidence to do so, and preregistered rules for which setup changes stop an attack missed their test on the protocol's three models. The paper releases .sova, an open-source tool and record format that audits verdicts, re-runs AgentDojo findings, and seals evidence for offline verification against published checksums.
Activity for this story
- Current comparable activity
- 6
- Peak comparable activity
- 10Nov 20 02:00
- Change in the past 24 hours
- –
Trends compare the same fully observed participants, which may cover fewer participants than current activity. Move the pointer or tap the chart to inspect hourly activity; use the left and right arrow keys with a keyboard.