CUGA Team Compares Three Agent Frameworks on the Same Model
Understand this story
The CUGA team used the same open-source model GLM-5.3-flash to compare three agent frameworks—CUGA, Hermes, and Deep Agents—on 90 AppWorld tasks, with task pass rates of 81.1%, 46.7%, and 30.0% respectively; the number of scenarios passed by all variants was 20, 8, and 2 respectively. The team states that this is a sample of 90 tasks on a single benchmark, and the results may not generalize to other tasks; the three configurations are not entirely identical, with CUGA using an AppWorld configuration the other frameworks lack; the experiments did not enable policies, so they do not measure policy execution effectiveness. AppWorld is a multi-step task benchmark covering 9 apps and 457 APIs.
AI overview based on source reports · Updated 3 days ago
Coverage timeline
Follow the coverage to explore different perspectives.
- Hugging Face BlogFeaturedSame Model, Different Harness: CUGA, Hermes, and Deep Agents — When to Pick CUGA?
IBM ran CUGA, Hermes, and Deep Agents on the same open-weight model (GLM-5.3-flash) across 90 AppWorld tasks, and task completion ranged from 30% to 81% depending only on the harness, with CUGA at 81.1% (73/90) versus Hermes at 46.7% and Deep Agents at 30.0%. CUGA is IBM's Apache 2.0 harness for enterprise API work, adding runtime-enforced policies (intent guards, tool approvals, ToolGuards, output formatting) and deployment inside sovereign or air-gapped environments. The authors note the 90-task sample, non-identical setups, and absence of policy testing as limits, and say CUGA is not the right fit for terminal coding or personal chat assistants.
Activity for this story
- Current comparable activity
- 3
- Peak comparable activity
- 10Nov 4 01:00
- Change in the past 24 hours
- -49%
Trends compare the same fully observed participants, which may cover fewer participants than current activity. Move the pointer or tap the chart to inspect hourly activity; use the left and right arrow keys with a keyboard.