OpenRouter releases an LLM benchmarking tutorial
Understand this story
OpenRouter has released an LLM benchmarking guide and accompanying scripts, advocating running the same tasks with the application's own real input and output formats, then comparing accuracy, billed cost, and request latency. In its own test of 60 synthetic tasks, GPT-6.1 Sol passed all 180/180 across three repetitions, while DeepSeek had the lowest response cost and passed 157/180; OpenRouter said the results had a Spearman correlation of only 0.109 with public rankings. The tool performs only offline validation by default; live options and quotas must be explicitly enabled to call models, and paid runs require the --live flag. The standard-library teaching runner, JSON examples, and comparisons passed offline checks on Windows x64 and Linux, but the accompanying full research sandbox and native Ori environment support only Windows x64.
AI overview based on source reports · Updated 3 days ago
Coverage timeline
Follow the coverage to explore different perspectives.
- OpenRouter AnnouncementsHow to Benchmark LLMs
A guide to benchmarking LLMs by writing tasks with script-checkable answers, grading with code, and comparing models on identical tasks via OpenRouter using graded accuracy, billed cost, and request time. On 60 synthetic tasks, GPT-6.1 Sol passed 180/180 trials while DeepSeek had the lowest response cost at 157/180 passes, and agreement with the public Artificial Analysis ranking was weak (Spearman correlation 0.109). The article includes a downloadable Python companion package with offline checks, configuration examples, and guidance on repeats, missing answers, and run comparison.
Activity for this story
- Current comparable activity
- 3
- Peak comparable activity
- 10Nov 12 09:00
- Change in the past 24 hours
- -49%
Trends compare the same fully observed participants, which may cover fewer participants than current activity. Move the pointer or tap the chart to inspect hourly activity; use the left and right arrow keys with a keyboard.