Browser Use evaluation adapter truncates answers, inflating scores
Understand this story
After Browser Use fixed the evaluation adapter running Claude Code, its agent evaluation score rose by 5.1 points; but a subsequent review found that the grading model only read 433 of the answer's 2,102 characters, and the answer had been truncated. Browser Use said that at most half of this round's increase came from flaws in its own evaluation process, rather than from improved model capability. The evaluation is based on BU Bench v2.1, whose tasks and grader have been made public in the benchmark repository.
AI overview based on source reports · Updated 51 min ago
Coverage timeline
Follow the coverage to explore different perspectives.
- Browser Use BlogOur Eval Said +5.1. Up to Half of It Was Our Own Bug.
After fixing the adapter that runs Claude Code in their agent eval, the authors saw a +5.1-point gain on 200 BU Bench v2.1 tasks (paired 95% CI [+1.0, +9.2]), but discovered the judge had only been reading the text after "FINAL ANSWER:" — about 433 of 2,102 median characters — so on 188 of 200 tasks it never saw the actual deliverable. Rejudging the saved baseline runs with full answers moved the baseline +2.6 points, and against a +0.8 no-change rejudge control the bug was worth +1.8, meaning a third to a half of the headline came from their own bug while the agent change earned +2.5 with an interval including zero. The post closes with eight rules, including rejudging baselines from saved evidence, treating judge errors as missing data rather than zeros, and comparing only runs with matching config hashes, task sets and runners.
Activity for this story
Not enough continuous observations to show a trend yet.