Computer-use agent database diff benchmark released
Understand this story
Author shubhamjoshipro releases the software-world-model benchmark, in which computer-use agents perform tasks in the browser interfaces of Invoice Ninja, Dolibarr, Odoo 17, and Odoo 18. Each episode starts from the same database snapshot, and after it ends a diff is taken across the entire database; outcomes are classified into four types: success, collateral damage, irreversible change, and false completion. Running it requires Docker, Python 3.10 to 3.12 (or automatic acquisition via uv), and about 8 GB of free disk space. All services bind to 127.0.0.1, so agents cannot leave localhost and cannot send real emails. The author states that in Phase 0, GPT-6 Luna cost 1 to 3 cents per successful task; the raw step logs and screenshots (about 800 MB) are not in the repository.
AI overview based on source reports · Updated 1 hr ago
Coverage timeline
Follow the coverage to explore different perspectives.
- Hacker News AIShow HN: A benchmark that diffs the whole database after an AI agent's task
This project benchmarks computer-use agents on self-hosted business apps (Invoice Ninja, Dolibarr, Odoo 17, Odoo 18) by starting each episode from an identical database snapshot and diffing the entire database afterward, yielding four verdicts: success, collateral damage, irreversible change, and false completion. In Phase 0 (635 scored episodes), Odoo 17 to 18 task success held (78% to 76%) while collateral damage rose from 15% to 21% and irreversible changes from 6% to 10%, and a 3-step composite task succeeded 0 of 50 times across 4 apps and 2 agents. The repo includes a Docker-based harness, per-app adapters, published episode summaries, and a documented custom-agent interface.
Activity for this story
Not enough continuous observations to show a trend yet.