How to Benchmark LLMs

A guide to benchmarking LLMs by writing tasks with script-checkable answers, grading with code, and comparing models on identical tasks via OpenRouter using graded accuracy, billed cost, and request time. On 60 synthetic tasks, GPT-6.1 Sol passed 180/180 trials while DeepSeek had the lowest response cost at 157/180 passes, and agreement with the public Artificial Analysis ranking was weak (Spearman correlation 0.109). The article includes a downloadable Python companion package with offline checks, configuration examples, and guidance on repeats, missing answers, and run comparison.