iCreatAI News
Skip to content
METR Blog·· 13 hr agoFeaturedAI score67

AI R&D-relevant model capabilities are improving rapidly and are getting harder to measure

AI R&D-relevant model capabilities are improving rapidly and are getting harder to measure

AI summary

METR reports that public frontier models up to Opus 5.5 and GPT-6 Astra usually match or outperform recruited skilled humans on its coding and optimization tasks, with time horizons on the combined TH1.1 and TH2 suite doubling roughly every 3.6 months, though models show weaker judgment than human experts on tasks lacking clear empirical feedback. The post also details methodology changes, including fewer human baselines (directly used for only 9% of time estimates), multiple score thresholds per task, and a new IRT-based METR Capability Index, while cautioning that containerized evals may overestimate real-world capabilities because they resemble RL training environments.

Why it matters

It offers a detailed, self-critical update on how frontier AI capabilities relevant to AI R&D acceleration are measured, including trend estimates and caveats about eval contamination.

Source: METR Blog · metr.org