DocBlocks releases docbench, a local small-model document benchmark
Understand this story
The DocBlocks team has released docbench, a document-task benchmark that uses 102 test cases, weighted across six categories of document tasks, to evaluate 24 local models ranging from 0.8B to 35B. Qwen 3.8 27B Q4 ranks first with 81 points, while Gemma 4 12B is in the same tier with 79 points; the best models at 4B and below score about 71 points, and cloud models score between 87 and 91. The evaluation shows that writing tasks are generally easy, review tasks have the greatest discriminative power, and small models often give trivial or incorrect suggestions. According to DocBlocks, the evaluation focuses on small models under 12B parameters; all models run on the same 23GB memory machine with llama.cpp and a 32K context, and the test documents are all in English. Scores are comparable only within docbench version 0.2; new versions will re-measure all models.
AI overview based on source reports · Updated 1 days ago
Coverage timeline
Follow the coverage to explore different perspectives.
- Hacker News AIBenchmarks for document tasks for local AI models
DocBlocks added support for local models to power document features and created "docbench," a 100-point weighted benchmark scoring 24 local models (0.8B to 35B parameters) on review, writing, diagrams, slides, study aids and transcript cleanup, with Qwen 3.8 27B Q4 leading at 81 and Gemma 4 12B statistically level at 79. Cloud models scored 87–91, at least 6 points above the best local model, and review separated models most: large models find real problems while 1–2B models pad reviews with trivial or wrong suggestions. The post recommends picking the highest-scoring model your machine can hold, calling Gemma 4 12B the sweet spot.
Activity for this story
- Current comparable activity
- 4
- Peak comparable activity
- 10Nov 17 04:00
- Change in the past 24 hours
- -51%
Trends compare the same fully observed participants, which may cover fewer participants than current activity. Move the pointer or tap the chart to inspect hourly activity; use the left and right arrow keys with a keyboard.