Hacker News AI· VictorSh·· 49 min agoAI score64
Uncovering 100+ errors in AI’s most popular benchmarks
Uncovering 100+ errors in AI’s most popular benchmarks
AI summary
Beaker autonomously flagged more than 100 errors in task instructions and grading across five widely used AI benchmarks, with at least one error verified in each. Examples include a grader accepting 5.9 as an exact match for 5 in Agents' Last Exam, nine email checks for a task asking agents to set up office hours in AutomationBench, and a Harvey LAB criterion expecting $0 where the source spreadsheet records $8 million. Dataset review compares task instructions, grading rules, and source files, flags mismatches, cites the evidence, and suggests corrections.
Source: Hacker News AI · withbeaker.ai