Vals AI Benchmark: Open-Source Models' Cyberattack Capabilities Approach Those of Closed-Source Models
Understand this story
Vals AI has released the CyberOffense benchmark, in which AI agents attack a simulated enterprise network from scratch on an isolated Kali Linux machine. GPT-6 Sol from OpenAI's restricted Daybreak project completed 80.8% of the objectives, the best result; the freely available open-weight model DeepSeek V4.1 Flash 1017 reached 72.8%, at a cost of about $3 per run, compared with about $22 for GPT-6 Sol. Vals researchers say that once model weights are made public, no provider can control or restrict their use. The tests also show that most models need to process more than 100 million tokens before making substantial progress.
AI overview based on source reports · Updated 1 hr ago
Coverage timeline
Follow the coverage to explore different perspectives.
- The DecoderFeaturedThe gap between open and closed AI models in cyber capabilities keeps shrinking
Vals AI's CyberOffense benchmark, which lets AI agents attack simulated corporate networks from an isolated Kali Linux machine with only a public web address, found OpenAI's restricted GPT-6 Sol Daybreak leading at 80.8% of objectives while the open-weight Deepseek V4.1 Flash 1017 reached 72.8% at about $3 per run versus roughly $22. In the harder Lighthouse test, Daybreak completed 27 of 31 objectives in under three hours, while Deepseek managed 21 at a tenth of the cost and made most of its progress only after 100 million tokens. Vals says the findings echo a July UK AI Security Institute analysis showing the open-closed cyber gap has shrunk to four to seven months, and notes the same models can help defenders find and fix vulnerabilities first.
Activity for this story
Not enough continuous observations to show a trend yet.