AI model cost comparison tool released, find the lowest price by benchmark score
Understand this story
Jeffery Kline has released an interactive page that uses data envelopment analysis (DEA) to compare the API cost per 1,000 benchmark tasks and the scores of various AI models, identifying the lowest-priced model or combination of models that does not lose points on any benchmark. Kline notes on the page that the savings here are only benchmark-based estimates; latency, quotas, switching, and other deployment costs are not included; splitting real work in this way has not yet been tested, and users should test alternatives on their own tasks. The costs shown on the page are estimates, the interface numbers are rounded, and the calculations use full precision.
AI overview based on source reports · Updated 3 days ago
Coverage timeline
Follow the coverage to explore different perspectives.
- Hacker News AIShow HN: A tool to optimize AI model cost without benchmark score trade-offs
This interactive page applies data envelopment analysis (following Zhou et al., 2022) to compare AI models' API cost per 1,000 benchmark tasks against their benchmark scores, identifying cheaper models or blends that match a given score. It notes that savings are benchmark estimates excluding latency, quotas, switching and other deployment costs, and that users should test alternatives on their own tasks. The single-slide creation task uses the free disposal hull variant and compares single models only, since its preference ratings cannot be averaged.
Activity for this story
- Current comparable activity
- 3
- Peak comparable activity
- 10Nov 3 22:00
- Change in the past 24 hours
- -51%
Trends compare the same fully observed participants, which may cover fewer participants than current activity. Move the pointer or tap the chart to inspect hourly activity; use the left and right arrow keys with a keyboard.