Show HN: A benchmark that diffs the whole database after an AI agent's task
Show HN: A benchmark that diffs the whole database after an AI agent's task
This project benchmarks computer-use agents on self-hosted business apps (Invoice Ninja, Dolibarr, Odoo 17, Odoo 18) by starting each episode from an identical database snapshot and diffing the entire database afterward, yielding four verdicts: success, collateral damage, irreversible change, and false completion. In Phase 0 (635 scored episodes), Odoo 17 to 18 task success held (78% to 76%) while collateral damage rose from 15% to 21% and irreversible changes from 6% to 10%, and a 3-step composite task succeeded 0 of 50 times across 4 apps and 2 agents. The repo includes a Docker-based harness, per-app adapters, published episode summaries, and a documented custom-agent interface.
Source: Hacker News AI · github.com