CloudOpsBench: AI cloud operations highest resolution rate 60.6%
Understand this story
The Ang Chen team at the University of Michigan released the open-source benchmark CloudOpsBench 1.0, designed to evaluate the ability of AI agents to perform real cloud operations tasks. The benchmark contains 45 AWS DevOps tasks, built by experienced DevOps engineers, with task results consistent between the real AWS cloud and a simulator; the project is open-sourced on GitHub under the Apache 2.0 license and includes detailed failure analysis, with GCP, Azure, and OCI versions planned for later release. In the published evaluations, Claude Fable 5.1 completed 60%, and GPT-6 Astra completed 53%. Previous reports stated that the highest resolution rate of 60.6% was achieved by claude-fable-5-1 (max) paired with claude-code, with a single run taking 38.6 hours, consuming 97.9M tokens, and costing $404.3. The team states that reasoning is the biggest bottleneck in AI cloud operations. Based on current results, the strongest combination still fails to solve nearly 40% of tasks.
AI overview based on source reports · Updated 2 days ago
Coverage timeline
Follow the coverage to explore different perspectives.
- The DecoderNew open-source benchmark reveals reasoning as the biggest bottleneck for AI in cloud operations
CloudOpsBench is a new open-source benchmark with 45 AWS tasks, built by experienced DevOps engineers, that measures how well AI agents handle typical cloud DevOps work such as fixing services, removing resources, restricting access rights, rolling out updates, and decommissioning projects. Even the best models fail many tasks: Claude Fable 5.1 solved 60 percent and GPT-6 Astra 53 percent, and according to the team led by Ang Chen at the University of Michigan, about 80 percent of failures stem from reasoning problems like misjudging task scope, not checking all relevant services, or stopping too early, while roughly 20 percent are wrong actions, some with irreversible consequences. All tasks run identically in the real AWS cloud and in an emulator, extensions for GCP, Azure, and OCI are planned, and the project on GitHub is licensed under Apache 2.0 with a detailed failure analysis also available.
- Hacker News AICloudOpsBench: AI benchmark shows FAI models struggling with cloud infra
CloudOpsBench is a benchmark for AI agents on real cloud operations, and its leaderboard ranks models by resolution rate, harness, release date, time, tokens and cost. The top entry, claude-fable-5-1 (max) with claude-code, reaches a 60.6% resolution rate at 38.6h, 97.9M tokens and $404.3, while the lowest listed, kimi-k3.1-pro (high) with terminus-2, reaches 19.1%.
Activity for this story
- Current comparable activity
- 4
- Peak comparable activity
- 13Nov 5 01:00
- Change in the past 24 hours
- -50%
Trends compare the same fully observed participants, which may cover fewer participants than current activity. Move the pointer or tap the chart to inspect hourly activity; use the left and right arrow keys with a keyboard.