By ByteBot
1Password’s security research unit just published the most clear-eyed study of AI-assisted vulnerability patching to date — and the numbers are not kind. Off-By-1-Labs tested 6,080 patches from ChatGPT-5.5 and Claude Opus 4.8 across six real-world CVEs. The headline: AI generates a clean, complete fix just 26% of the time. The other 74% produces what researchers call FLAWED patches — Fix-Like Artifacts With Embedded Defects that either fail to close the vulnerability, break application behavior, or introduce a new security flaw.
The Numbers, Without Spin
Off-By-1-Labs generated 540 patches per CVE per model, then evaluated each against three criteria: did it fully eliminate the vulnerability, did it preserve application behavior, and did it avoid introducing new risks? The results:
- 26.0% — clean fix, no side effects
- 53.9% — FLAWED: unresolved vulnerability, altered behavior, or a new vulnerability added
- 20.1% — fixed the CVE but materially changed how the application behaves
- ~5% — introduced a brand-new vulnerability on top of failing to fix the original
Keith Hoodlet, Director of Security Research at 1Password, puts it plainly: “The expected value of a fully LLM-generated, non-human-reviewed patch is a net-negative by a considerable margin.” These are not mid-tier models being tested here. ChatGPT-5.5 and Claude Opus 4.8 represent the current ceiling of frontier AI capability. If the best models available get it right one in four times, the industry’s confidence in AI-generated patches is running well ahead of the actual results.
The Fragile Fix: The Failure Mode That Looks Like Success
The most dangerous finding is not the 53.9% outright failure rate — that is at least visible. The real threat is what researchers call the fragile fix: a patch that passes tests, survives code review, and gets merged, but does not actually address the underlying vulnerability.
More than one-third of patches classified as “successful” were flagged as fragile. The definition: a fix that gates the known exploit path behind a targeted check while leaving the vulnerable code intact and reachable via a different call path.
The SpringAI SpEL injection CVE makes the problem concrete. Both ChatGPT-5.5 and Claude Opus 4.8 generated patches that escaped specific characters in user input — the exact characters used in the proof-of-concept exploit. The PoC stopped working. The Spring Expression Language injection vulnerability that made the PoC possible remained completely untouched. The day someone finds a different character sequence, the fix is worthless.
This points to a fundamental framing problem. A human security engineer reads a PoC exploit as evidence of a vulnerability class. An LLM reads it as the specific input that fails, and patches exactly that input. The model generates the statistically most plausible fix — which is almost always the narrowest possible fix.
What the CVEs Exposed
The study tested six real CVEs spanning Linux, Apache, Exim, and SpringAI. Three examples illustrate the pattern:
- CVE-2026-31431 “Copy Fail” (Linux kernel algif_aead, CVSS 7.8): Models patched the specific copy path that triggered the privilege escalation but left adjacent surfaces in the same crypto API module exposed.
- CVE-2026-45185 (Exim, CVSS 9.8, unauthenticated RCE): A multi-step exploit chain. Models consistently patched one link in the chain while leaving the others intact.
- GHSA-wpqr-6v78-jr5g (Gemini CLI RCE): AI models were asked to patch an RCE in an AI CLI tool. The patch success rate was no better than the study average.
A separate study from the University of San Francisco, the Vector Institute, and UMass Boston adds a compounding concern: after five rounds of AI-driven patch refinement, the number of critical vulnerabilities in the tested codebases rose 37.6%. More AI iteration does not solve AI patching failures.
What DevSecOps Teams Should Do Now
The research is not an argument against using AI in security workflows. It is an argument against removing humans from the final verification step. Off-By-1-Labs recommends three concrete changes:
- Execution-grounded verification: Run the original exploit against the candidate patch. Do not rely on code inspection alone. The fragile fix problem exists precisely because fragile patches look correct in review.
- Required security engineer sign-off: AI-generated patches need a mandatory human review gate before merge — separate from and in addition to automated test pipelines.
- Treat AI patches as first drafts: AI surfaces the relevant code and generates candidate fixes quickly. That draft is the starting point for a security engineer, not the finishing point.
The broader DevSecOps toolchain has caught on. Vendors including Checkmarx, Snyk, and Cycode now flag AI-generated code as a distinct risk category requiring its own scan layer. If your pipeline does not separate AI-generated patch validation from standard code review, it should.
The Bottom Line
AI is a capable first-pass at vulnerability patching. It is not a trusted last word. 1Password’s research gives the DevSecOps community what it needed: hard numbers from frontier models, not vendor promises. The 26% success figure belongs on the wall of every security team considering removing human review from AI patch workflows. The fragile fix problem belongs there too — because a patch that passes CI but leaves the vulnerability code intact is not a patch. It is a liability that has not been discovered yet.
