iCreatAI News
Skip to content
The Decoder· Jonathan Kemper·· 40 min agoAI score70

AI coding agents ace isolated tests but complete fewer than one in ten tasks in a simulated workplace

AI coding agents ace isolated tests but complete fewer than one in ten tasks in a simulated workplace

AI summary

AI coding agents are approaching top scores on standard benchmarks, but a new experiment placed them as digital employees in a simulated software company with colleagues, project boards, and chat histories. Even Claude Opus 5.5 and GPT-6 Astra fully completed fewer than one in ten tasks. The piece notes Anthropic's claim that Claude writes over 80 percent of its production code and that benchmarks like SWE-bench are nearly saturated.

Source: The Decoder · the-decoder.com