Artificial Analysis releases Endpoint Accuracy Index 1.1

1. 01:08
   Hacker News AI

   1 in 4 3rd Party AI Endpoints Underperform Their Advertised Models

   Artificial Analysis released version 1.1 of its Endpoint Accuracy Index, which compares more than 100 serverless endpoints hosting open-weight models against reference deployments of the official weights, finding roughly one in four score below their reference. The index reruns each endpoint on BFCL v4-500, HLE-250 and AA-LCR-25, and version 1.1 raises long-context repeats from 10 to 25, cutting the average 95% confidence interval from plus or minus 4.7 to plus or minus 2.6 points, while coverage expands to six models including DeepSeek V4.1 Flash 1017, GLM-5.3, GLM-5.3 Flash, Muse Spark Open, Kimi K3 and gpt-oss-120b. Reported gaps trace to output caps that truncate answers, shorter reasoning, and a schema bug encoding string tool arguments as numbers, with GLM-5.3 endpoints scoring from 67.2% to 99.3% and all 22 DeepSeek endpoints statistically consistent with the reference.

Trends compare the same fully observed participants, which may cover fewer participants than current activity. Move the pointer or tap the chart to inspect hourly activity; use the left and right arrow keys with a keyboard.