iCreatAI News
Skip to content
Trending storyUpdating

Contextflo experiment: four notes boost AI agent accuracy

1 reports1 sources17 hr agoUpdated

Understand this story

AI overview

Contextflo has published an experiment testing context notes for AI agents on a payments dataset. On the Adyen and Hugging Face DABstep dataset, the same Claude Haiku 5.5 agent faced 42 held-out questions: given only the data tables, it answered 40% correctly; adding company documentation actually lowered accuracy to 36%; adding four business notes then raised it to 76%, while cost per run fell from about 5 cents to about 1.3 cents. Contextflo says the experiment tested only this one dataset and did not test larger-scale databases, so it does not claim the results scale; the four notes were a deliberately set upper limit. The dataset can be reused for free (CC BY 4.0) and loaded as-is.

AI overview based on source reports · Updated 16 hr ago

Coverage timeline

Follow the coverage to explore different perspectives.

Nov 20
  1. Hacker News AI
    Show HN: Company docs didn't help our AI agent. Four shortnotes doubled accuracy

    Running a Claude Haiku 5.5 agent on the DABstep payments dataset, the authors found company docs alone gave no accuracy gain over tables alone (36% vs 40% of tries correct on a held-out test), though they cut cost. Four short notes written from the agent's wrong answers raised accuracy to 76% at about a quarter of the cost per answer, and the gain held on 42 questions the notes were never written from.

Activity for this story

Current comparable activity
6
Peak comparable activity
10Nov 20 04:00
Change in the past 24 hours
–

Trends compare the same fully observed participants, which may cover fewer participants than current activity. Move the pointer or tap the chart to inspect hourly activity; use the left and right arrow keys with a keyboard.