New open-source benchmark reveals reasoning as the biggest bottleneck for AI in cloud operations

CloudOpsBench is a new open-source benchmark with 45 AWS tasks, built by experienced DevOps engineers, that measures how well AI agents handle typical cloud DevOps work such as fixing services, removing resources, restricting access rights, rolling out updates, and decommissioning projects. Even the best models fail many tasks: Claude Fable 5.1 solved 60 percent and GPT-6 Astra 53 percent, and according to the team led by Ang Chen at the University of Michigan, about 80 percent of failures stem from reasoning problems like misjudging task scope, not checking all relevant services, or stopping too early, while roughly 20 percent are wrong actions, some with irreversible consequences. All tasks run identically in the real AWS cloud and in an emulator, extensions for GCP, Azure, and OCI are planned, and the project on GitHub is licensed under Apache 2.0 with a detailed failure analysis also available.