Contextflo experiment: four notes boost AI agent accuracy
Understand this story
Contextflo has published an experiment testing context notes for AI agents on a payments dataset. On the Adyen and Hugging Face DABstep dataset, the same Claude Haiku 5.5 agent faced 42 held-out questions: given only the data tables, it answered 40% correctly; adding company documentation actually lowered accuracy to 36%; adding four business notes then raised it to 76%, while cost per run fell from about 5 cents to about 1.3 cents. Contextflo says the experiment tested only this one dataset and did not test larger-scale databases, so it does not claim the results scale; the four notes were a deliberately set upper limit. The dataset can be reused for free (CC BY 4.0) and loaded as-is.
AI overview based on source reports · Updated 16 hr ago
Coverage timeline
Follow the coverage to explore different perspectives.
- Hacker News AIShow HN: Company docs didn't help our AI agent. Four shortnotes doubled accuracy
Running a Claude Haiku 5.5 agent on the DABstep payments dataset, the authors found company docs alone gave no accuracy gain over tables alone (36% vs 40% of tries correct on a held-out test), though they cut cost. Four short notes written from the agent's wrong answers raised accuracy to 76% at about a quarter of the cost per answer, and the gain held on 42 questions the notes were never written from.
Activity for this story
- Current comparable activity
- 6
- Peak comparable activity
- 10Nov 20 04:00
- Change in the past 24 hours
- –
Trends compare the same fully observed participants, which may cover fewer participants than current activity. Move the pointer or tap the chart to inspect hourly activity; use the left and right arrow keys with a keyboard.