Vals AI and NextLadder release legal aid AI benchmark; none of 23 models reach the 80% reliability line
Understand this story
On November 10, Vals AI and NextLadder Ventures released the Access to Justice benchmark, evaluating 23 AI models using 252 simulated legal aid conversations. The results showed that none of the models reached the 80% reliable first-contact threshold set by the organizers. The highest score was 76.4% for Alibaba's Qwen 3.8 Max. In the evaluation, refusals to answer caused by content filtering were counted as zero points.
AI overview based on source reports · Updated 27 min ago
Coverage timeline
Follow the coverage to explore different perspectives.
- Hacker News AINew 'Access to Justice' Benchmark: Leading AI Models Miss Critical Legal Details
Vals AI and NextLadder Ventures released the Access to Justice Benchmark on November 10, testing 23 AI models on 252 simulated legal-aid conversations, with none reaching the 80% threshold for a reliable first point of contact and top performer Alibaba's Qwen 3.8 Max scoring 76.4%. Models met 78–87% of criteria for general legal explanations but only 30–50% for identifying specific local resources, with 21 of 23 falling back on national directories in at least 70% of final answers. Free-tier models averaged 69.7% versus 70.4% for premium ones, and housing questions scored highest at 75.4% while public benefits ranked lowest at 57.3%.
Activity for this story
Not enough continuous observations to show a trend yet.