mlsanityData to 7 Oct 2026

Automated Program Repair: Emerging trends pose and expose problems for benchmarks

arXiv 2405.05455 ↗First posted 8 May 2024cs.SE cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

Machine learning (ML) now pervades the field of Automated Program Repair (APR). Algorithms deploy neural machine translation and large language models (LLMs) to generate software patches, among other tasks. But, there are important differences between these applications of ML and earlier work. Evaluations and comparisons must take care to ensure that results are valid and likely to generalize. A challenge is that the most popular APR evaluation benchmarks were not designed with ML techniques in mind. This is especially true for LLMs, whose large and often poorly-disclosed training datasets may include problems on which they are evaluated.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 69 of its 136 references that are in the map. First posted on arXiv 8 May 2024.

Builds on

69 of 136 references.

References · 69

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2021Program Synthesis with Large Language Modelscited by 3,509
  10. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  11. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  12. 2020Extracting Training Data from Large Language Modelscited by 2,144
38 more
  1. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  2. 2019A Survey on Bias and Fairness in Machine Learningcited by 1,911
  3. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  4. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,303
  5. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  6. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,024
  7. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  8. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  9. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 515
  10. 2019Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classificationcited by 396
  11. 2021Unified Pre-training for Program Understanding and Generationcited by 361
  12. 2022A Systematic Evaluation of Large Language Models of Codecited by 360
  13. 2018Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifierscited by 341
  14. 2022Language Models of Code are Few-Shot Commonsense Learnerscited by 160
  15. 2022What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?cited by 149
  16. 2018SequenceR: Sequence-to-Sequence Learning for End-to-End Program Repaircited by 144
  17. 2018The Adverse Effects of Code Duplication in Machine Learning Models of Codecited by 123
  18. 2019An Attentive Survey of Attention Modelscited by 123
  19. 2018An Empirical Study on Learning Bug-Fixing Patches in the Wild via Neural Machine Translationcited by 106
  20. 2021CURE: Code-Aware Neural Machine Translation for Automatic Program Repaircited by 85
  21. 2020Improved Code Summarization via a Graph Neural Networkcited by 79
  22. 2019TBar: Revisiting Template-based Automated Program Repaircited by 64
  23. 2019Do We Train on Test Data? Purging CIFAR of Near-Duplicatescited by 61
  24. 2021A Syntax-Guided Edit Decoder for Neural Program Repaircited by 57
  25. 2022PUMA: Performance Unchanged Model Augmentation for Training Data Removalcited by 56
  26. 2021Neural Program Repair with Execution-based Backpropagationcited by 50
  27. 2022CoditT5: Pretraining for Source Code and Natural Language Editingcited by 45
  28. 2020Deflating Dataset Bias Using Synthetic Data Augmentationcited by 37
  29. 2021Neural Transfer Learning for Repairing Security Vulnerabilities in C Codecited by 36
  30. 2020Pre-training Is (Almost) All You Need: An Application to Commonsense Reasoningcited by 33
  31. 2022SelfAPR: Self-supervised Program Repair with Test Execution Diagnosticscited by 33
  32. 2017ARJA: Automated Repair of Java Programs via Multi-Objective Genetic Programmingcited by 26
  33. 2018FixMiner: Mining Relevant Fix Patterns for Automated Program Repaircited by 22
  34. 2020Evaluating Representation Learning of Code Changes for Predicting Patch Correctness in Program Repaircited by 22
  35. 2021Trust Enhancement Issues in Program Repaircited by 22
  36. 2020Patching as Translation: the Data and the Metaphorcited by 18
  37. 2020On the Efficiency of Test Suite based Program Repair: A Systematic Assessment of 16 Automated Repair Systems for Java Programscited by 16
  38. 2023Do We Train on Test Data? The Impact of Near-Duplicates on License Plate Recognitioncited by 14

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 3.

7 papers.

  1. 2024Benchmarks and Metrics for Evaluations of Code Generation: A Critical Reviewcited by 19
  2. 2025An Empirical Study on Failures in Automated Issue Solvingcited by 15
  3. 2026SVRepair: Structured Visual Reasoning for Automated Program Repaircited by 2
  4. 2025Collaborative Agents for Automated Program Repair in Rubycited by 1
  5. 2026Rethinking Automated Program Repair: The Impact of Bug Complexity, Fault Localization, and LLM Cost-efficiencycited by 1
  6. 2025Evaluating LLMs for One-Shot Patching of Real and Artificial Vulnerabilitiescited by 0
  7. 2026MultiFixer: A Coordinator-Proposer Based Multi-Agent Framework For Fixing Multi-Hunk Bugscited by 0