iCreatAI News
Skip to content
Trending storyUpdating

Computer-use agent database diff benchmark released

1 reports1 sources2 hr agoUpdated

Understand this story

AI overview

Author shubhamjoshipro releases the software-world-model benchmark, in which computer-use agents perform tasks in the browser interfaces of Invoice Ninja, Dolibarr, Odoo 17, and Odoo 18. Each episode starts from the same database snapshot, and after it ends a diff is taken across the entire database; outcomes are classified into four types: success, collateral damage, irreversible change, and false completion. Running it requires Docker, Python 3.10 to 3.12 (or automatic acquisition via uv), and about 8 GB of free disk space. All services bind to 127.0.0.1, so agents cannot leave localhost and cannot send real emails. The author states that in Phase 0, GPT-6 Luna cost 1 to 3 cents per successful task; the raw step logs and screenshots (about 800 MB) are not in the repository.

AI overview based on source reports · Updated 1 hr ago

Coverage timeline

Follow the coverage to explore different perspectives.

Nov 16
  1. Hacker News AI
    Show HN: A benchmark that diffs the whole database after an AI agent's task

    This project benchmarks computer-use agents on self-hosted business apps (Invoice Ninja, Dolibarr, Odoo 17, Odoo 18) by starting each episode from an identical database snapshot and diffing the entire database afterward, yielding four verdicts: success, collateral damage, irreversible change, and false completion. In Phase 0 (635 scored episodes), Odoo 17 to 18 task success held (78% to 76%) while collateral damage rose from 15% to 21% and irreversible changes from 6% to 10%, and a 3-step composite task succeeded 0 of 50 times across 4 apps and 2 agents. The repo includes a Docker-based harness, per-app adapters, published episode summaries, and a documented custom-agent interface.

Activity for this story

Not enough continuous observations to show a trend yet.