# Benchmarking Automated Security Patch Backporting: How Far Are We?

arXiv: https://arxiv.org/abs/2608.17671 · HTML: https://mlsanity.com/paper/2608.17671/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 60 references that are in the map.
- First posted on arXiv 18 Aug 2026.

## Builds on (most cited first, 15 of 15)

- [CVEfixes: Automated Collection of Vulnerabilities and Their Fixes from Open-Source Software](https://mlsanity.com/paper/2107.08760/) (2021)
- [TBar: Revisiting Template-based Automated Program Repair](https://mlsanity.com/paper/1903.08409/) (2019)
- [A Survey of LLM-based Automated Program Repair: Taxonomies, Design Paradigms, and Applications](https://mlsanity.com/paper/2506.23749/) (2025)
- [Hybrid Automated Program Repair by Combining Large Language Models and Program Analysis](https://mlsanity.com/paper/2406.00992/) (2024)
- [APPATCH: Automated Adaptive Prompting Large Language Models for Real-World Software Vulnerability Patching](https://mlsanity.com/paper/2408.13597/) (2024)
- [SoK: Towards Effective Automated Vulnerability Repair](https://mlsanity.com/paper/2501.18820/) (2025)
- [Repair Ingredients Are All You Need: Improving Large Language Model-Based Program Repair via Repair Ingredients Search](https://mlsanity.com/paper/2506.23100/) (2025)
- [Patch-to-PoC: A Systematic Study of Agentic LLM Systems for Linux Kernel N-Day Reproduction](https://mlsanity.com/paper/2602.07287/) (2026)
- [An Investigation of Patch Porting Practices of the Linux Kernel Ecosystem](https://mlsanity.com/paper/2402.05212/) (2024)
- [VADER: A Human-Evaluated Benchmark for Vulnerability Assessment, Detection, Explanation, and Remediation](https://mlsanity.com/paper/2505.19395/) (2025)
- [Automating Zero-Shot Patch Porting for Hard Forks](https://mlsanity.com/paper/2404.17964/) (2024)
- [Patch Validation in Automated Vulnerability Repair](https://mlsanity.com/paper/2603.06858/) (2026)
- [MigGPT: Harnessing Large Language Models for Automated Migration of Out-of-Tree Linux Kernel Patches Across Versions](https://mlsanity.com/paper/2504.09474/) (2025)
- [PortGPT: Towards Automated Backporting Using Large Language Models](https://mlsanity.com/paper/2510.22396/) (2025)
- [BackportBench: A Multilingual Benchmark for Automated Backporting of Patches](https://mlsanity.com/paper/2512.01396/) (2025)

## Abstract (arXiv)

Automated security patch backporting is critical for mitigating N-day vulnerabilities. Recent tools report success rates above 80% on their respective datasets. However, these evaluations are often confined to homogeneous environments, such as one repository or specific project versions. Consequently, it remains unclear how well these tools generalize beyond their originally targeted scenarios. We present Porting Benchmark, a curated dataset of 1,234 security patch backporting cases spanning cross-version, cross-branch, and cross-repository scenarios, paired with a common evaluation framework. Using this benchmark, we evaluate five tools spanning program analysis, LLM prompting, and LLM agents under aligned settings. Our results show that aligned evaluation changes the apparent performance landscape: PortGPT and TSBPort remain comparatively strong on the Replication Dataset, while FixMorph and Mystique degrade substantially under the common protocol. Performance degrades sharply on structurally complex patches: the best commit-level success rate falls from 85.2% on Type-I patches to 24.0% on Type-IV. We identify four root-cause categories (missing target API awareness, cross-version semantic mismatch, non-local dependency propagation failure, and patch construction or localization failure) and derive concrete directions for next-generation tool design. On a 45-case dynamically validated subset with verified test cases and constructed POCs, we further observe that reference-based benchmark scores do not fully capture real-world remediation: exact match sharply under-credits harder target adaptations, while executable validation reveals residual integration failures in the target that static reference agreement misses. Executable-feedback refinement provides limited but measurable recovery on the hardest executable cases.