# Applying CodeBERT for Automated Program Repair of Java Simple Bugs

arXiv: https://arxiv.org/abs/2103.11626 · HTML: https://mlsanity.com/paper/2103.11626/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Mar 2021.

## Built on by (most cited first, 30 of 44)

- [ReACC: A Retrieval-Augmented Code Completion Framework](https://mlsanity.com/paper/2203.07722/) (2022)
- [CVEfixes: Automated Collection of Vulnerabilities and Their Fixes from Open-Source Software](https://mlsanity.com/paper/2107.08760/) (2021)
- [Is ChatGPT the Ultimate Programming Assistant -- How far is it?](https://mlsanity.com/paper/2304.11938/) (2023)
- [How Effective Are Neural Networks for Fixing Security Vulnerabilities](https://mlsanity.com/paper/2305.18607/) (2023)
- [Large Language Models and Simple, Stupid Bugs](https://mlsanity.com/paper/2303.11455/) (2023)
- [A Survey on Machine Learning Techniques for Source Code Analysis](https://mlsanity.com/paper/2110.09610/) (2021)
- [CigaR: Cost-efficient Program Repair with LLMs](https://mlsanity.com/paper/2402.06598/) (2024)
- [AutoPruner: Transformer-Based Call Graph Pruning](https://mlsanity.com/paper/2209.03230/) (2022)
- [VulCurator: A Vulnerability-Fixing Commit Detector](https://mlsanity.com/paper/2209.03260/) (2022)
- [Who Evaluates the Evaluators? On Automatic Metrics for Assessing AI-based Offensive Code Generators](https://mlsanity.com/paper/2212.06008/) (2022)
- [Detect-Localize-Repair: A Unified Framework for Learning to Debug with CodeT5](https://mlsanity.com/paper/2211.14875/) (2022)
- [Large Language Model Evaluation Via Multi AI Agents: Preliminary results](https://mlsanity.com/paper/2404.01023/) (2024)
- [Automating Code-Related Tasks Through Transformers: The Impact of Pre-training](https://mlsanity.com/paper/2302.04048/) (2023)
- [Is this Change the Answer to that Problem? Correlating Descriptions of Bug and Code Changes for Evaluating Patch Correctness](https://mlsanity.com/paper/2208.04125/) (2022)
- [Investigating the Performance of Language Models for Completing Code in Functional Programming Languages: a Haskell Case Study](https://mlsanity.com/paper/2403.15185/) (2024)
- [Improving Code Autocompletion with Transfer Learning](https://mlsanity.com/paper/2105.05991/) (2021)
- [DexBERT: Effective, Task-Agnostic and Fine-grained Representation Learning of Android Bytecode](https://mlsanity.com/paper/2212.05976/) (2022)
- [FlakyFix: Using Large Language Models for Predicting Flaky Test Fix Categories and Test Code Repair](https://mlsanity.com/paper/2307.00012/) (2023)
- [Automating the Correctness Assessment of AI-generated Code for Security Contexts](https://mlsanity.com/paper/2310.18834/) (2023)
- [Improving Stack Overflow question title generation with copying enhanced CodeBERT model and bi-modal information](https://mlsanity.com/paper/2109.13073/) (2021)
- [Multi-Granularity Detector for Vulnerability Fixes](https://mlsanity.com/paper/2305.13884/) (2023)
- [ConDefects: A New Dataset to Address the Data Leakage Concern for LLM-based Fault Localization and Program Repair](https://mlsanity.com/paper/2310.16253/) (2023)
- [AI for DevSecOps: A Landscape and Future Opportunities](https://mlsanity.com/paper/2404.04839/) (2024)
- [Test2Vec: An Execution Trace Embedding for Test Case Prioritization](https://mlsanity.com/paper/2206.15428/) (2022)
- [Using Developer Discussions to Guide Fixing Bugs in Software](https://mlsanity.com/paper/2211.06335/) (2022)
- [Enhancing Robustness of AI Offensive Code Generators via Data Augmentation](https://mlsanity.com/paper/2306.05079/) (2023)
- [Enhancing AI-based Generation of Software Exploits with Contextual Information](https://mlsanity.com/paper/2408.02402/) (2024)
- [LecPrompt: A Prompt-based Approach for Logical Error Correction with CodeBERT](https://mlsanity.com/paper/2410.08241/) (2024)
- [Diverse Title Generation for Stack Overflow Posts with Multiple Sampling Enhanced Transformer](https://mlsanity.com/paper/2208.11523/) (2022)
- [Automated Code Fix Suggestions for Accessibility Issues in Mobile Apps](https://mlsanity.com/paper/2408.03827/) (2024)

## Abstract (arXiv)

Software debugging, and program repair are among the most time-consuming and labor-intensive tasks in software engineering that would benefit a lot from automation. In this paper, we propose a novel automated program repair approach based on CodeBERT, which is a transformer-based neural architecture pre-trained on large corpus of source code. We fine-tune our model on the ManySStuBs4J small and large datasets to automatically generate the fix codes. The results show that our technique accurately predicts the fixed codes implemented by the developers in 19-72% of the cases, depending on the type of datasets, in less than a second per bug. We also observe that our method can generate varied-length fixes (short and long) and can fix different types of bugs, even if only a few instances of those types of bugs exist in the training dataset.