Our Eval Said +5.1. Up to Half of It Was Our Own Bug.
Our Eval Said +5.1. Up to Half of It Was Our Own Bug.
After fixing the adapter that runs Claude Code in their agent eval, the authors saw a +5.1-point gain on 200 BU Bench v2.1 tasks (paired 95% CI [+1.0, +9.2]), but discovered the judge had only been reading the text after "FINAL ANSWER:" — about 433 of 2,102 median characters — so on 188 of 200 tasks it never saw the actual deliverable. Rejudging the saved baseline runs with full answers moved the baseline +2.6 points, and against a +0.8 no-change rejudge control the bug was worth +1.8, meaning a third to a half of the headline came from their own bug while the agent change earned +2.5 with an interval including zero. The post closes with eight rules, including rejudging baselines from saved evidence, treating judge errors as missing data rather than zeros, and comparing only runs with matching config hashes, task sets and runners.
Source: Browser Use Blog · browser-use.com