iCreatAI News
Skip to content
Trending storyUpdating

Browser Use evaluation adapter truncates answers, inflating scores

1 reports1 sources21 hr agoUpdated

Understand this story

AI overview

After Browser Use fixed the evaluation adapter running Claude Code, its agent evaluation score rose by 5.1 points; but a subsequent review found that the grading model only read 433 of the answer's 2,102 characters, and the answer had been truncated. Browser Use said that at most half of this round's increase came from flaws in its own evaluation process, rather than from improved model capability. The evaluation is based on BU Bench v2.1, whose tasks and grader have been made public in the benchmark repository.

AI overview based on source reports · Updated 51 min ago

Coverage timeline

Follow the coverage to explore different perspectives.

Nov 20
  1. Browser Use Blog
    Our Eval Said +5.1. Up to Half of It Was Our Own Bug.

    After fixing the adapter that runs Claude Code in their agent eval, the authors saw a +5.1-point gain on 200 BU Bench v2.1 tasks (paired 95% CI [+1.0, +9.2]), but discovered the judge had only been reading the text after "FINAL ANSWER:" — about 433 of 2,102 median characters — so on 188 of 200 tasks it never saw the actual deliverable. Rejudging the saved baseline runs with full answers moved the baseline +2.6 points, and against a +0.8 no-change rejudge control the bug was worth +1.8, meaning a third to a half of the headline came from their own bug while the agent change earned +2.5 with an interval including zero. The post closes with eight rules, including rejudging baselines from saved evidence, treating judge errors as missing data rather than zeros, and comparing only runs with matching config hashes, task sets and runners.

Activity for this story

Not enough continuous observations to show a trend yet.