Research team releases First Silicon chip defect benchmark
Understand this story
The research team umwelt.vision has released the First Silicon hardware defect detection benchmark, designed to test whether frontier models can find bugs in real chip designs. The team collected 86 engineer-fixed cases from open-source chip designs, 84 of which are real bugs; during testing, the model is not told which version it is seeing, and a formal checker then determines whether the SystemVerilog assertions written by the model hit the mark. The benchmark publicly discloses the full contents of only 5 bugs, keeping the remaining 81 private so that future models can continue to use it; each bug and prompt is accompanied by a salted SHA-256 commitment, with the salt revealed during audits. Claude models run as read-only, tool-free Claude Code subagents, with each prompt run only once. The team says other models can also be evaluated.
AI overview based on source reports · Updated 2 days ago
Coverage timeline
Follow the coverage to explore different perspectives.
- Hacker News AIShow HN: First Silicon, real chip bugs that AI models miss
First Silicon is a benchmark of 86 engineer fixes in open-source chip designs (84 real bugs, 70 from OpenTitan) in which models write SystemVerilog assertions judged by Yosys and SymbiYosys on both buggy and fixed versions. The best configuration, Claude Opus 5.5 at maximum effort, caught 16 of 86 cases with zero false alarms, and 67 of the 84 genuine fixes were caught cleanly by none of the four models at default settings. Letting Opus 5.5 call the bounded model checker did not close the gap, as the model dropped or changed 79% of counterexampled assertions, including 8 that had correctly exposed a bug.
Activity for this story
- Current comparable activity
- 4
- Peak comparable activity
- 10Nov 5 02:00
- Change in the past 24 hours
- -50%
Trends compare the same fully observed participants, which may cover fewer participants than current activity. Move the pointer or tap the chart to inspect hourly activity; use the left and right arrow keys with a keyboard.