# PATCHEVAL: A New Benchmark for Evaluating LLMs on Patching Real-World Vulnerabilities

arXiv: https://arxiv.org/abs/2511.11019 · HTML: https://mlsanity.com/paper/2511.11019/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Nov 2025.

## Built on by (most cited first, 21 of 21)

- [CVE-Factory: Scaling Expert-Level Agentic Tasks for Code Security Vulnerability](https://mlsanity.com/paper/2602.03012/) (2026)
- [SEC-bench Pro: Can Language Models Solve Long-Horizon Software Security Tasks?](https://mlsanity.com/paper/2605.26548/) (2026)
- [ZeroDayBench: Evaluating LLM Agents on Unseen Zero-Day Vulnerabilities for Cyberdefense](https://mlsanity.com/paper/2603.02297/) (2026)
- [FrogNano: Training a 4B Coding Agent via Online Task Synthesis](https://mlsanity.com/paper/2609.07925/) (2026)
- [EvoRepair: Enhancing Vulnerability Repair Agents Through Experience-Based Self-Evolution](https://mlsanity.com/paper/2605.30105/) (2026)
- [Self-Evolving Coding Agents](https://mlsanity.com/paper/2608.03392/) (2026)
- [PatchBench: Evaluating AI Agents for Vulnerability Patching](https://mlsanity.com/paper/2609.04075/) (2026)
- [A Dual-Loop Agent Framework for Automated Vulnerability Reproduction](https://mlsanity.com/paper/2602.05721/) (2026)
- [SecCodeBench-V2 Technical Report](https://mlsanity.com/paper/2602.15485/) (2026)
- [Why LLMs Fail: A Failure Analysis and Partial Success Measurement for Automated Security Patch Generation](https://mlsanity.com/paper/2603.10072/) (2026)
- [SEVRA-BENCH: Social Engineering of Vulnerabilities in Review Agents](https://mlsanity.com/paper/2606.13757/) (2026)
- [Beyond Refusal: A Same-Lineage Study of Aligned and Abliterated LLMs for Vulnerability Analysis](https://mlsanity.com/paper/2607.05842/) (2026)
- [Knowledge-Enhanced Agentic Vulnerability Repair](https://mlsanity.com/paper/2607.00820/) (2026)
- [From Runnable to Verifiable: An Independent Reproducibility Study of LLM/Agent-Driven Vulnerability Validation Artifacts](https://mlsanity.com/paper/2608.09567/) (2026)
- [Activation Probes Surface Code-Security Signals that the Model's Output Misses](https://mlsanity.com/paper/2608.09643/) (2026)
- [WeSCE: A Benchmark for Measuring Security Drift in LLM-Driven Code Editing](https://mlsanity.com/paper/2608.15092/) (2026)
- [Evaluating Multiple LLM Generations with Validated Task Coverage](https://mlsanity.com/paper/2608.24228/) (2026)
- [SecureVibe: Making Vibe Coding More Secure](https://mlsanity.com/paper/2609.38606/) (2026)
- [AuraForge: Scaling Security Supervision for Training Coding Agents](https://mlsanity.com/paper/2610.00850/) (2026)
- [CWE-bench: Refocusing Agentic Security from Exploitation to Defense](https://mlsanity.com/paper/2610.17020/) (2026)
- [P2Patch: Exploit-Guided Vulnerability Repair](https://mlsanity.com/paper/2610.20245/) (2026)

## Abstract (arXiv)

Software vulnerabilities are increasing at an alarming rate. However, manual patching is both time-consuming and resource-intensive, while existing automated vulnerability repair (AVR) techniques remain limited in effectiveness. Recent advances in large language models (LLMs) have opened a new paradigm for AVR, demonstrating remarkable progress. To examine the capability of LLMs in AVR, several vulnerability benchmarks have been proposed recently. However, they still suffer from key limitations of outdated vulnerabilities, limited language coverage, unreliable patch validation, and insufficient reproducibility. To overcome these challenges, we introduce PATCHEVAL, a multilingual benchmark for Go, JavaScript, and Python, languages for which existing benchmarks remain unexplored. PATCHEVAL curates a dataset of 1,000 vulnerabilities drawn from CVEs reported between 2015 and 2025, covering 65 distinct CWEs. A subset of 230 CVEs is further equipped with runtime sandbox environments, enabling patch verification through both security tests and functionality tests. To provide a systematic comparison of LLM-based vulnerability repair, we evaluate a series of state-of-the-art LLMs and agents, presenting an in-depth analysis that empirically yields key insights to guide future research in AVR.