SemiAnalysis releases GDPx agent benchmark, with the highest score at only 12%
Understand this story
SemiAnalysis releases the GDPx benchmark, placing AI agents in simulated work environments rather than testing their knowledge. Its 206 tasks span 11 domains including chip design, biology, quantitative trading, cybersecurity, and law, and no model scores above 12%. In the tasks, agents have up to 8 hours to operate professional software such as Siemens NX, search data rooms containing thousands of files, or attack real enterprise systems. The total score is calculated as pass@1 over three attempts, with each domain weighted equally, and when a domain has multiple partners, each partner is weighted equally. SemiAnalysis plans to update the benchmark every three to six months; typically each partner publicly releases only one sample task.
AI overview based on source reports · Updated 8 hr ago
Coverage timeline
Follow the coverage to explore different perspectives.
- The DecoderNo AI agent scores above 12 percent on SemiAnalysis' new real-world work benchmark
SemiAnalysis released GDPx, a benchmark of 206 tasks across eleven domains that places AI agents in simulated work environments operating professional software for up to eight hours. Claude Opus 5.5 leads at 12 percent, followed by Gemini 4 Carbon at 10.9 percent and GPT-6 Astra at 9.6 percent, and no model fully solves the finance and legal tasks. Critics including developer Josh Purtell call GDPx "more noise than signal," citing provider-specific agent software and small sub-domain task counts.
- SemiAnalysisGDPx: Measuring Frontier Model Capabilities Across the Most Valuable Domains
SemiAnalysis introduces GDPx, a benchmark covering domains such as chip design, cybersecurity, biology, finance, legal, quant trading, RSI, hardware design, materials science, alignment, and software engineering, built with 14 specialized data partners; the best model scores just 12%. The article argues OpenAI's GDPval is inadequate for today's agents because its tasks are non-interactive and lack realistic environments, and it details GDPx's QA process, per-domain task design, and model results including safety-classifier effects.
Activity for this story
- Current comparable activity
- 7
- Peak comparable activity
- 10Nov 6 01:00
- Change in the past 24 hours
- –
Trends compare the same fully observed participants, which may cover fewer participants than current activity. Move the pointer or tap the chart to inspect hourly activity; use the left and right arrow keys with a keyboard.