iCreatAI News
Skip to content
Trending storyUpdating

Study says AI task-length predictions are overestimated

1 reports1 sources21 hr agoUpdated

Understand this story

AI overview

A study based on real internal usage data from OpenAI and Google points out that the task length corresponding to an 80% success rate for AI on real research tasks is about 9 minutes for Google and 16 minutes for OpenAI after adjustment. The study says extrapolated estimates from ECI to METR range from 33 to 41 times, and the original AI 2027 scenario overestimates by more than 100 times. The conclusion concerns the deviation between the aforementioned real task lengths and the forecast charts.

AI overview based on source reports · Updated 8 hr ago

Coverage timeline

Follow the coverage to explore different perspectives.

Nov 21
  1. Hacker News AI
    The Most Important Charts in AI Are Wrong

    Real-world data from OpenAI and Google shows that AI models complete internal work tasks of only about 9–16 minutes at 80% success, far below the 30–40 hours or months projected by METR/ECI extrapolations and the AI 2027 scenario, with the gap widening over time. The author argues that AI is more capable than reliable, that high-reliability task lengths remain short, and that the estimated self-improvement loop (about 3.6% faster experimentation per ECI point) stays below the 15–19% threshold for self-sustaining recursive self-improvement. The piece calls for greater transparency from AI companies, especially Anthropic, which it says forced METR to redact key evidence.

Activity for this story

Current comparable activity
6
Peak comparable activity
10Nov 21 11:00
Change in the past 24 hours
–

Trends compare the same fully observed participants, which may cover fewer participants than current activity. Move the pointer or tap the chart to inspect hourly activity; use the left and right arrow keys with a keyboard.