# BackportBench: A Multilingual Benchmark for Automated Backporting of Patches

arXiv: https://arxiv.org/abs/2512.01396 · HTML: https://mlsanity.com/paper/2512.01396/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 67 references that are in the map.
- First posted on arXiv 1 Dec 2025.

## Builds on (most cited first, 29 of 29)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://mlsanity.com/paper/2407.16741/) (2024)
- [BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions](https://mlsanity.com/paper/2406.15877/) (2024)
- [CodeBLEU: a Method for Automatic Evaluation of Code Synthesis](https://mlsanity.com/paper/2009.10297/) (2020)
- [Agentless: Demystifying LLM-based Software Engineering Agents](https://mlsanity.com/paper/2407.01489/) (2024)
- [Large Language Model-Based Agents for Software Engineering: A Survey](https://mlsanity.com/paper/2409.02977/) (2024)
- [Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving](https://mlsanity.com/paper/2504.02605/) (2025)
- [Large Language Model-Brained GUI Agents: A Survey](https://mlsanity.com/paper/2411.18279/) (2024)
- [MAGIS: LLM-Based Multi-Agent Framework for GitHub Issue Resolution](https://mlsanity.com/paper/2403.17927/) (2024)
- [SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents](https://mlsanity.com/paper/2406.12952/) (2024)
- [SWE-Search: Enhancing Software Agents with Monte Carlo Tree Search and Iterative Refinement](https://mlsanity.com/paper/2410.20285/) (2024)
- [A Survey on Code Generation with LLM-based Agents](https://mlsanity.com/paper/2508.00083/) (2025)
- [Learn-by-interact: A Data-Centric Framework for Self-Adaptive Agents in Realistic Environments](https://mlsanity.com/paper/2501.10893/) (2025)
- [Trae Agent: An LLM-based Agent for Software Engineering with Test-time Scaling](https://mlsanity.com/paper/2507.23370/) (2025)
- [Agents in Software Engineering: Survey, Landscape, and Vision](https://mlsanity.com/paper/2409.09030/) (2024)
- [SWE-bench-java: A GitHub Issue Resolving Benchmark for Java](https://mlsanity.com/paper/2408.14354/) (2024)
- [Enhancing repository-level software repair via repository-aware knowledge graphs](https://mlsanity.com/paper/2503.21710/) (2025)
- [A Survey of LLM-based Automated Program Repair: Taxonomies, Design Paradigms, and Applications](https://mlsanity.com/paper/2506.23749/) (2025)
- [OmniGIRL: A Multilingual and Multimodal Benchmark for GitHub Issue Resolution](https://mlsanity.com/paper/2505.04606/) (2025)
- [CoCoEvo: Co-Evolution of Programs and Test Cases to Enhance Code Generation](https://mlsanity.com/paper/2502.10802/) (2025)
- [CODEMENV: Benchmarking Large Language Models on Code Migration](https://mlsanity.com/paper/2506.00894/) (2025)
- [MigrationBench: Repository-Level Code Migration Benchmark from Java 8](https://mlsanity.com/paper/2505.09569/) (2025)
- [Do Software Security Practices Yield Fewer Vulnerabilities?](https://mlsanity.com/paper/2210.14884/) (2022)
- [I depended on you and you broke me: An empirical study of manifesting breaking changes in client packages](https://mlsanity.com/paper/2301.04563/) (2023)
- [Mitigating Persistence of Open-Source Vulnerabilities in Maven Ecosystem](https://mlsanity.com/paper/2308.03419/) (2023)
- [Automating Zero-Shot Patch Porting for Hard Forks](https://mlsanity.com/paper/2404.17964/) (2024)
- [An Empirical Study on Package-Level Deprecation in Python Ecosystem](https://mlsanity.com/paper/2408.10327/) (2024)
- [Tracing Vulnerabilities in Maven: A Study of CVE lifecycles and Dependency Networks](https://mlsanity.com/paper/2502.04621/) (2025)

## Built on by (most cited first, 1 of 1)

- [Benchmarking Automated Security Patch Backporting: How Far Are We?](https://mlsanity.com/paper/2608.17671/) (2026)

## Abstract (arXiv)

Many modern software projects evolve rapidly to incorporate new features and security patches. It is important for users to update their dependencies to safer versions, but many still use older, vulnerable package versions because upgrading can be difficult and may break their existing codebase. Software developers can mitigate this problem by backporting security patches to older releases. However, manually backporting is time-consuming and error-prone. The effectiveness of existing automated backporting techniques on general software remains unclear since they typically target only code-hunk or function-level patch porting scenarios and are evaluated with imperfect metrics. To facilitate the development and evaluation of automated backporting techniques, we introduce BackportBench, the first comprehensive benchmark suite for patch backporting problem. BackportBench is a multilingual benchmark that contains 202 patch backporting problems from PyPI, Maven, and npm, each with executable Docker environments and relevant test cases. We evaluated existing patch porting methods and LLM-based techniques that have the potential to adapt to this task using BackportBench. The results show that the agentic method has outperformed traditional patch porting methods, especially on cases that require logical and structural changes. However, the performance varies across different programming languages. Based on the findings, we draw several implications for researchers and software practitioners in future work on automated backporting.