CWE-bench v2 tests security fixes that preserve working code

What changed in version two

Collinear AI’s 3 November announcement describes 100 public tasks covering 91 Common Weakness Enumeration categories. Agents receive an open-source repository and an instruction. The authors say scoring is programmatic, tasks were manually reviewed, and a separate held-out set remains available for evaluation.

The published comparison puts Gemini 4 Argon at 67.8% pass@1 and Claude Opus 5.5 at 67.3%. Those are reported benchmark results, not our measurements. The release also compares multiple attempts and rollout costs. Public tasks, graders and a harness allow others to examine how the evaluation works.

The small gap between the leading reported single-attempt scores should not be treated as a complete purchasing decision. The task set, agent configuration, number of attempts and cost all shape the result. Source: CWE-bench v2: More coverage, quality, and transparency

Analysis: patch correctness has more than one dimension

The useful distinction here is between removing the observed failure and preserving the application’s intended contract. Consider an illustrative file-upload service. Disabling uploads could prevent an unsafe upload, but it would not be an acceptable repair if legitimate uploads are a required feature. A good evaluation needs to test both sides.

Programmatic checks make a result more inspectable, but the checks still define what counts as success. A test can miss a behavior that matters in production or overconstrain a solution that would otherwise be valid. Public access creates an opportunity for scrutiny rather than a guarantee that the evaluation is complete.

Pass@1 and pass@4 describe different operating conditions. A workflow that can afford multiple candidates, a test environment and a selection step is different from one that accepts the first patch. Teams should compare a model under the retry policy they can actually operate. Additional attempts need to be considered alongside the additional time and review burden.

Cost per rollout is only one component of that decision. If reviewers spend substantial time identifying incomplete repairs, cheap generation may not produce a cheap accepted fix. Conversely, a more expensive model may not justify its price if the simpler model already passes the organization’s relevant cases.

How to use the release without overreading it

A practical evaluation should first identify which weakness classes resemble the organization’s own code and review needs. Run the published harness in an isolated environment, preserve the configuration and inspect a sample of successes and failures. A reproducible score without an explanation of failure modes is less useful to the engineers deciding what to trust.

Then add private regression cases reflecting the application’s contracts. These should remain separate from any public leaderboard score. Mixing them into an unnamed aggregate would obscure whether an improvement came from general security-repair ability or familiarity with the team’s patterns.

Teams should also preserve the original task instructions and record any extra hints given to the agent. A result obtained after a reviewer points directly at the faulty function describes a different level of assistance from an unaided repair. Both may have business value, but they should be labeled separately. That makes a benchmark-informed trial useful for choosing a workflow rather than merely choosing a model name.

The most valuable output is a map of where the agent helps and where it requires intervention: locating the defect, proposing a narrow patch, preserving behavior or interpreting a failed test. That map can guide supervised deployment. CWE-bench v2’s public design is useful because it allows claims to be challenged. It should encourage more careful inspection of AI-generated patches, not replace it.