Open-source fraud triage AI decision API comparison benchmark
Understand this story
Smart Mobile House has open-sourced fraud-decision-lab, which compares the payment fraud ranking performance of TypeSafe Jev, OpenAI Decisions, Cloudflare Clef/Clef-flash, and structured-output LLMs versus XGBoost on labeled tabular data. The project is positioned as a vendor-neutral research benchmark comparing specialized AI decision models with traditional machine learning on fraud ranking. By default, the project runs without API keys, dataset downloads, or paid calls; to connect to online providers, users must supply API keys, enable --allow-external-requests, and provide a --max-cost-usd cost estimate. This estimate is not a billing cap enforced by the provider, and tokenization, retries, and billing uncertainty may all cause actual costs to differ. The built-in IEEE-CIS configuration is restricted to local runs and does not permit external processing.
AI overview based on source reports · Updated 2 days ago
Coverage timeline
Follow the coverage to explore different perspectives.
- Hacker News AIOpen benchmark of AI decision APIs vs. XGBoost for fraud triage
Smart Mobile House released fraud-decision-lab, a vendor-neutral benchmark that sends the same transactions to TypeSafe Jev, OpenAI Decisions, Cloudflare Clef/Clef-flash and a structured-output LLM, then measures how much labelled fraud each ranks into the top 1% and 5% of an analyst review queue against an XGBoost baseline, with cost and latency. It runs offline by default on bundled synthetic data, saves predictions and experiment metadata for local verification, and requires API keys, --allow-external-requests and a --max-cost-usd estimate for live provider runs. The project notes that XGBoost learns from labelled history while the AI models score transactions zero-shot, that the bundled data is synthetic, and that the current source distribution contains no benchmark result files.
Activity for this story
- Current comparable activity
- 3
- Peak comparable activity
- 10Nov 13 01:00
- Change in the past 24 hours
- -50%
Trends compare the same fully observed participants, which may cover fewer participants than current activity. Move the pointer or tap the chart to inspect hourly activity; use the left and right arrow keys with a keyboard.