# RepairLLaMA: Efficient Representations and Fine-Tuned Adapters for Program Repair

arXiv: https://arxiv.org/abs/2312.15698 · HTML: https://mlsanity.com/paper/2312.15698/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 72 references that are in the map.
- First posted on arXiv 25 Dec 2023.

## Builds on (most cited first, 30 of 40)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [StarCoder: may the source be with you!](https://mlsanity.com/paper/2305.06161/) (2023)
- [InCoder: A Generative Model for Code Infilling and Synthesis](https://mlsanity.com/paper/2204.05999/) (2022)
- [Efficient Training of Language Models to Fill in the Middle](https://mlsanity.com/paper/2207.14255/) (2022)
- [OctoPack: Instruction Tuning Code Large Language Models](https://mlsanity.com/paper/2308.07124/) (2023)
- [SequenceR: Sequence-to-Sequence Learning for End-to-End Program Repair](https://mlsanity.com/paper/1901.01808/) (2018)
- [On the Effectiveness of Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2211.15583/) (2022)
- [Full Parameter Fine-tuning for Large Language Models with Limited Resources](https://mlsanity.com/paper/2306.09782/) (2023)
- [InferFix: End-to-End Program Repair with LLMs](https://mlsanity.com/paper/2303.07263/) (2023)
- [Graph-based, Self-Supervised Program Repair from Diagnostic Feedback](https://mlsanity.com/paper/2005.10636/) (2020)
- [CURE: Code-Aware Neural Machine Translation for Automatic Program Repair](https://mlsanity.com/paper/2103.00073/) (2021)
- [Impact of Code Language Models on Automated Program Repair](https://mlsanity.com/paper/2302.05020/) (2023)
- [Less Training, More Repairing Please: Revisiting Automated Program Repair via Zero-shot Learning](https://mlsanity.com/paper/2207.08281/) (2022)
- [Repair Is Nearly Generation: Multilingual Program Repair with LLMs](https://mlsanity.com/paper/2208.11640/) (2022)
- [A Syntax-Guided Edit Decoder for Neural Program Repair](https://mlsanity.com/paper/2106.08253/) (2021)
- [Copiloting the Copilots: Fusing Large Language Models with Completion Engines for Automated Program Repair](https://mlsanity.com/paper/2309.00608/) (2023)
- [LLaMA-Reviewer: Advancing Code Review Automation with Large Language Models through Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2308.11148/) (2023)
- [CoditT5: Pretraining for Source Code and Natural Language Editing](https://mlsanity.com/paper/2208.05446/) (2022)
- [RAP-Gen: Retrieval-Augmented Patch Generation with CodeT5 for Automatic Program Repair](https://mlsanity.com/paper/2309.06057/) (2023)
- [No More Fine-Tuning? An Experimental Evaluation of Prompt Tuning in Code Intelligence](https://mlsanity.com/paper/2207.11680/) (2022)
- [Neural Transfer Learning for Repairing Security Vulnerabilities in C Code](https://mlsanity.com/paper/2104.08308/) (2021)
- [SelfAPR: Self-supervised Program Repair with Test Execution Diagnostics](https://mlsanity.com/paper/2203.12755/) (2022)
- [A Critical Review of Large Language Model on Software Engineering: An Example from ChatGPT and Automated Program Repair](https://mlsanity.com/paper/2310.08879/) (2023)
- [Exploring Parameter-Efficient Fine-Tuning Techniques for Code Generation with Large Language Models](https://mlsanity.com/paper/2308.10462/) (2023)
- [Towards Efficient Fine-tuning of Pre-trained Code Models: An Experimental Study and Beyond](https://mlsanity.com/paper/2304.05216/) (2023)
- [CIRCLE: Continual Repair across Programming Languages](https://mlsanity.com/paper/2205.10956/) (2022)
- [One Adapter for All Programming Languages? Adapter Tuning for Code Search and Summarization](https://mlsanity.com/paper/2303.15822/) (2023)

## Built on by (most cited first, 30 of 50)

- [When LLMs Meet Cybersecurity: A Systematic Literature Review](https://mlsanity.com/paper/2405.03644/) (2024)
- [Large Language Models for Cyber Security: A Systematic Literature Review](https://mlsanity.com/paper/2405.04760/) (2024)
- [A Survey on LoRA of Large Language Models](https://mlsanity.com/paper/2407.11046/) (2024)
- [CTIBench: A Benchmark for Evaluating LLMs in Cyber Threat Intelligence](https://mlsanity.com/paper/2406.07599/) (2024)
- [CodeUltraFeedback: An LLM-as-a-Judge Dataset for Aligning Large Language Models to Coding Preferences](https://mlsanity.com/paper/2403.09032/) (2024)
- [Can LLMs Replace Manual Annotation of Software Engineering Artifacts?](https://mlsanity.com/paper/2408.05534/) (2024)
- [Low-Rank Adaptation for Foundation Models: A Comprehensive Review](https://mlsanity.com/paper/2501.00365/) (2024)
- [Llama-3.1-FoundationAI-SecurityLLM-Base-8B Technical Report](https://mlsanity.com/paper/2504.21039/) (2025)
- [SWE-Synth: Synthesizing Verifiable Bug-Fix Data to Enable Large Language Models in Resolving Real-World Bugs](https://mlsanity.com/paper/2504.14757/) (2025)
- [From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligence](https://mlsanity.com/paper/2511.18538/) (2025)
- [SECURE: Benchmarking Large Language Models for Cybersecurity](https://mlsanity.com/paper/2405.20441/) (2024)
- [COAST: Enhancing the Code Debugging Ability of LLMs through Communicative Agent Based Data Synthesis](https://mlsanity.com/paper/2408.05006/) (2024)
- [Collu-Bench: A Benchmark for Predicting Language Model Hallucinations in Code](https://mlsanity.com/paper/2410.09997/) (2024)
- [Are Large Language Models Memorizing Bug Benchmarks?](https://mlsanity.com/paper/2411.13323/) (2024)
- [Seeing is Fixing: Cross-Modal Reasoning with Multimodal LLMs for Visual Software Issue Fixing](https://mlsanity.com/paper/2506.16136/) (2025)
- [RepairBench: Leaderboard of Frontier Models for Program Repair](https://mlsanity.com/paper/2409.18952/) (2024)
- [SGAgent: Suggestion-Guided LLM-Based Multi-Agent Framework for Repository-Level Software Repair](https://mlsanity.com/paper/2602.23647/) (2026)
- [Enhancing Automated Program Repair with Solution Design](https://mlsanity.com/paper/2408.12056/) (2024)
- [The Art of Repair: Optimizing Iterative Program Repair with Instruction-Tuned Models](https://mlsanity.com/paper/2505.02931/) (2025)
- [RepoRepair: Leveraging Code Documentation for Repository-Level Automated Program Repair](https://mlsanity.com/paper/2603.01048/) (2026)
- [Towards Reliable Evaluation of Neural Program Repair with Natural Robustness Testing](https://mlsanity.com/paper/2402.11892/) (2024)
- [Memory-Efficient Large Language Models for Program Repair with Semantic-Guided Patch Generation](https://mlsanity.com/paper/2410.16655/) (2024)
- [Automated Cloud Infrastructure-as-Code Reconciliation with AI Agents](https://mlsanity.com/paper/2510.20211/) (2025)
- [Integrating Various Software Artifacts for Better LLM-based Bug Localization and Program Repair](https://mlsanity.com/paper/2412.03905/) (2024)
- [DefenderBench: A Toolkit for Evaluating Language Agents in Cybersecurity Environments](https://mlsanity.com/paper/2506.00739/) (2025)
- [Evaluating the Generalizability of LLMs in Automated Program Repair](https://mlsanity.com/paper/2503.09217/) (2025)
- [The Impact of Fine-tuning Large Language Models on Automated Program Repair](https://mlsanity.com/paper/2507.19909/) (2025)
- [Wisdom and Delusion of LLM Ensembles for Code Generation and Repair](https://mlsanity.com/paper/2510.21513/) (2025)
- [Distilling Desired Comments for Enhanced Code Review with Large Language Models](https://mlsanity.com/paper/2412.20340/) (2024)
- [Treefix: Enabling Execution with a Tree of Prefixes](https://mlsanity.com/paper/2501.12339/) (2025)

## Abstract (arXiv)

Automated Program Repair (APR) has evolved significantly with the advent of Large Language Models (LLMs). Fine-tuning LLMs for program repair is a recent avenue of research, with many dimensions which have not been explored. Existing work mostly fine-tune LLMs with naive code representations and does not scale to frontier models. To address this problem, we propose RepairLLaMA, a novel program repair approach that 1) identifies optimal code representations for APR with fine-tuned models, and 2) pioneers state-of-the-art parameter-efficient fine-tuning technique (PEFT) for program repair. This results in RepairLLaMA producing a highly effective `program repair adapter' for fixing bugs with AI. Our experiments demonstrate the validity of both concepts. First, fine-tuning adapters with program repair specific code representations enables the model to use meaningful repair signals and produce better patches. Second, parameter-efficient fine-tuning helps fine-tuning to converge and clearly contributes to the effectiveness of RepairLLaMA in fixing bugs outside the fine-tuning data distribution. Overall, RepairLLaMA correctly fixes 144 Defects4J v2, 109 HumanEval-Java, and 20 GitBug-Java bugs, outperforming all baselines.