Key points
- Gemini 4 Argon leads with 67.8% pass@1 on the 100-task CWE-bench v2.
- DeepSeek V4.1 Flash achieves 57.0% pass@1 at a cost of $0.04 per rollout.
- The benchmark covers 91 distinct Common Weakness Enumerations across open-source repos.
Collinear AI published CWE-bench v2 on Hugging Face, introducing a public set of 100 tasks to evaluate frontier models on security fixes.confirmed Gemini 4 Argon leads the leaderboard with a 67.8% pass@1 rate, narrowly ahead of Claude Opus 5.5 at 67.3%.Collinear AI has released a new tool to test how well AI models can fix security bugs in software code.Collinear AI’s CWE-bench v2 provides a deterministic, programmatic evaluation of frontier models on 100 security tasks across 91 CWEs. The benchmark tests agents on finding and fixing weaknesses in real open-source repositories without breaking existing functionality.
The figures
Gemini 4 Argon achieves 67.8% pass@1 and 81.0% pass@4. Claude Opus 5.5 records 67.3% pass@1 and 78.0% pass@4. DeepSeek V4.1 Flash reaches 57.0% pass@1 at $0.04 per rollout. Claude Opus 5.5 costs $1.83 per rollout, while Gemini 4 Argon costs $3.92. Of the 11 models evaluated, 95 tasks are solved by at least one model. The leader fails 19% of the set on every attempt.
Behind the figures
Each task is a self-contained Harbor environment with a deterministic verifier. The setup mirrors previous versions, providing an agent with a repository checkout and a short instruction. Scoring is fully programmatic, ensuring deterministic results that anyone can reproduce. LLM judge panels curated the checks to ensure hardness and fairness. Every task was reviewed by hand to verify that instructions are realistic and verifiers check for correct behaviour rather than specific fix structures.
What changes, what does not
The public set now includes 100 tasks covering 91 different CWEs. The setup remains the same as before, but the quality control process is explicitly documented. Verifiers are designed to fail memorised solutions, such as known upstream patches. A held-out set is maintained for tracking progress on unseen tasks, accessible only via contact with Collinear AI. The dataset is gated, requiring acceptance of access conditions on the Hugging Face page.
What builders can do with it
Developers can run the benchmark using Harbor, which fetches tasks directly from the Hugging Face repo without manual cloning. The `registry.json` file allows the harness to run the entire benchmark automatically. Tasks are tagged as `rl-environment`, meaning the same environments and verifiers can provide rewards for training defensive agents. Builders can evaluate their models on the public set and submit results via pull requests to the leaderboard.
Why it matters
- The benchmark provides a deterministic, programmatic method for evaluating model capabilities on security fixes, removing subjectivity from assessments.
- The Pareto frontier of cost and accuracy allows developers to choose between low-cost models like DeepSeek and high-accuracy models like Gemini.
- Coverage of 91 distinct CWEs ensures that models are tested on a wide range of vulnerability classes, not just common patterns.
- The held-out set prevents overfitting to the public benchmark, allowing for ongoing evaluation of model progress on unseen tasks.
Topics: security, benchmark, Collinear AI, Hugging Face, models
Updated
Then / now / next
Previous versions of CWE-bench were available, but the specific task count and public access details were not explicitly stated in this advisory.
CWE-bench v2 opens up a public set of 100 tasks with documented quality control and programmatic scoring.
Progress on the held-out set can be tracked by contacting Collinear AI, providing a long-term view of model improvements.
Story timeline
- Then Previous versions of CWE-bench were available, but the specific task count and public access details were not explicitly stated in this advisory.
- 03 Nov 2026 CWE-bench v2: More coverage, quality, and transparency
- Now CWE-bench v2 opens up a public set of 100 tasks with documented quality control and programmatic scoring.
- Next Progress on the held-out set can be tracked by contacting Collinear AI, providing a long-term view of model improvements.
For readers
Developers
- Run the benchmark using Harbor with the command `harbor run` and the `registry.json` file.
- Submit results to the leaderboard via a pull request on the model's repository.
Researchers
- Use the held-out set for ongoing evaluation by contacting [email protected].
- Analyse the Pareto frontier to understand the trade-offs between cost and accuracy.
Policy
- Review the deterministic scoring methods to understand how model capabilities are measured.
- Note that the benchmark covers 91 distinct CWEs, providing a broad scope for assessment.
Numbers in context
Circle area is proportional to the magnitude; labels retain signs. Figures may describe different measures. Read their contexts above.Try it
harbor run -k 4
Good result: The command runs four attempts per task and reports the mean reward over all 400 trials.
Ask the article
Local keyword search of this article; no generated answers.Did this article give you the evidence you needed?
Your choice stays on this device. No public tally.Terms in this story
pass@1
The percentage of tasks solved correctly on the first attempt.
pass@4
The percentage of tasks solved correctly within four attempts.
Harbor
A Dockerized environment system used to run the benchmark tasks.
CWE
Common Weakness Enumeration, a standard list of software security weakness types.
Questions readers ask
How can I evaluate my model on the held-out set?
You must contact [email protected] to evaluate your model on the held-out set, which is not publicly available.
What is the cost of running the benchmark on Gemini 4 Argon?
Gemini 4 Argon costs $3.92 per rollout, according to the Pareto frontier data in the advisory.
Can I use the tasks for training agents?
Yes, the tasks are tagged as `rl-environment`, allowing the use of environments and verifiers for providing rewards during training.
Connected stories
Reading receipt
- Gemini 4 Argon leads with 67.8% pass@1 on the 100-task CWE-bench v2.
- DeepSeek V4.1 Flash achieves 57.0% pass@1 at a cost of $0.04 per rollout.
- The benchmark covers 91 distinct Common Weakness Enumerations across open-source repos.
Still open: How do API-only models compare in terms of latency and throughput beyond the reported pass rates?
Still open: What are the specific instructions and verifiers for the held-out set?