AI R&D-relevant model capabilities are improving rapidly and are getting harder to measure
AI R&D-relevant model capabilities are improving rapidly and are getting harder to measure
METR reports that public frontier models up to Opus 5.5 and GPT-6 Astra usually match or outperform recruited skilled humans on its coding and optimization tasks, with time horizons on the combined TH1.1 and TH2 suite doubling roughly every 3.6 months, though models show weaker judgment than human experts on tasks lacking clear empirical feedback. The post also details methodology changes, including fewer human baselines (directly used for only 9% of time estimates), multiple score thresholds per task, and a new IRT-based METR Capability Index, while cautioning that containerized evals may overestimate real-world capabilities because they resemble RL training environments.
It offers a detailed, self-critical update on how frontier AI capabilities relevant to AI R&D acceleration are measured, including trend estimates and caveats about eval contamination.
Source: METR Blog · metr.org