iCreatAI News
Skip to content
Hacker News AI· VictorSh·· 49 min agoAI score64

Uncovering 100+ errors in AI’s most popular benchmarks

Uncovering 100+ errors in AI’s most popular benchmarks

AI summary

Beaker autonomously flagged more than 100 errors in task instructions and grading across five widely used AI benchmarks, with at least one error verified in each. Examples include a grader accepting 5.9 as an exact match for 5 in Agents' Last Exam, nine email checks for a task asking agents to set up office hours in AutomationBench, and a Harvey LAB criterion expecting $0 where the source spreadsheet records $8 million. Dataset review compares task instructions, grading rules, and source files, flags mismatches, cites the evidence, and suggests corrections.

Source: Hacker News AI · withbeaker.ai