Beaker says five AI benchmarks contain over a hundred errors
Understand this story
Beaker used an automated approach to review the task instructions and scoring rules of five commonly used AI benchmarks, flagging more than 100 errors and saying at least one error in each benchmark was verified. Examples Beaker listed include: Agents' Last Exam's grader treated 5.9 as an exact match for 5; all nine checks in AutomationBench targeted emails and did not verify whether office hours were created; Harvey LAB's scoring criteria required $0, while the source spreadsheet recorded $8 million. These errors involve the benchmarks' task descriptions and scoring steps and may affect the evaluation results of the relevant benchmarks.
AI overview based on source reports · Updated 15 min ago
Coverage timeline
Follow the coverage to explore different perspectives.
- Hacker News AIUncovering 100+ errors in AI’s most popular benchmarks
Beaker autonomously flagged more than 100 errors in task instructions and grading across five widely used AI benchmarks, with at least one error verified in each. Examples include a grader accepting 5.9 as an exact match for 5 in Agents' Last Exam, nine email checks for a task asking agents to set up office hours in AutomationBench, and a Harvey LAB criterion expecting $0 where the source spreadsheet records $8 million. Dataset review compares task instructions, grading rules, and source files, flags mismatches, cites the evidence, and suggests corrections.
Activity for this story
Not enough continuous observations to show a trend yet.