Vetto releases ScienceBench benchmark, strongest model achieves 49% pass rate

1. 01:15
   Hacker News AI

   Vetto ScienceBench: Expanding Scientific Capabilities of Frontier AI Models

   Vetto ScienceBench introduces 50 new data-analysis tasks in the Terminal-Bench Science format across 14 scientific disciplines, each reviewed for quality and audited per trial, where the best of 13 frontier models passes 49% of trials and most fail over 80%. Tasks require agents to explore data, choose a method, check its assumptions and deliver an answer or pipeline graded against held-out ground truth, with data features like instrument artifacts or selection effects that defeat textbook approaches. In about two thirds of failures the model had already seen the problem in its own output and then dismissed it as noise or corrected it with the wrong model, and a one-line warning raised GPT-6.1 Sol's pass rate from 29% to 39% without improving outcomes once the problem was noticed.

Trends compare the same fully observed participants, which may cover fewer participants than current activity. Move the pointer or tap the chart to inspect hourly activity; use the left and right arrow keys with a keyboard.