DocBlocks releases docbench, a local-model document task benchmark
Understand this story
The DocBlocks team has released docbench, using 102 test cases and a 100-point weighted scale to evaluate 24 local models ranging from 0.8B to 35B on six categories of document tasks: review, writing, charts, slides, study aids and transcription cleanup. The team says the evaluation focuses on small models with fewer than 12 billion parameters. The models run on llama.cpp, mostly using 4-bit quantization (Q4), with a context length of 32,768 tokens; all test documents are in English. Scoring does not include speed, so a well-writing slow model can still score high. The team also compared GPT 6.1 Sol, Claude Opus 5.5 and Claude Sonnet 5.5 as controls via command-line interfaces.
AI overview based on source reports · Updated 19 min ago
Coverage timeline
Follow the coverage to explore different perspectives.
- Hacker News AIBenchmarks for document tasks for local AI models
DocBlocks created "docbench," a 100-point weighted benchmark scoring 24 local models (0.8B–35B parameters) on six document-task families—review, writing, diagrams, slides, study aids and narration cleanup—using 102 test cases with automatic checks and AI judging. Qwen 3.8 27B Q4 led local models at 81/100, with Gemma 4 12B statistically level at 79 and the best 4B-and-under models reaching about 71, while cloud models scored 87–91. Review separated models most: large models found real problems, whereas 1–2B models padded reviews with trivial or wrong suggestions.
Activity for this story
Not enough continuous observations to show a trend yet.