mlsanityData to 14 Oct 2026

RepairBench: Leaderboard of Frontier Models for Program Repair

arXiv 2409.18952 ↗First posted 27 Sep 2024cs.SE cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

AI-driven program repair uses AI models to repair buggy software by producing patches. Rapid advancements in AI surely impact state-of-the-art performance of program repair. Yet, grasping this progress requires frequent and standardized evaluations. We propose RepairBench, a novel leaderboard for AI-driven program repair. The key characteristics of RepairBench are: 1) it is execution-based: all patches are compiled and executed against a test suite, 2) it assesses frontier models in a frequent and standardized way. RepairBench leverages two high-quality benchmarks, Defects4J and GitBug-Java, to evaluate frontier models against real-world program repair tasks. We publicly release the evaluation framework of RepairBench. We will update the leaderboard as new frontier models are released.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 31 of its 53 references that are in the map. First posted on arXiv 27 Sep 2024.

Builds on

31 of 53 references.

References · 31

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  7. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  8. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  9. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  10. 2024Qwen2.5-Coder Technical Reportcited by 1,203
  11. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  12. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 546
19 more
  1. 2024Agentless: Demystifying LLM-based Software Engineering Agentscited by 421
  2. 2022A Systematic Evaluation of Large Language Models of Codecited by 363
  3. 2022The Stack: 3 TB of permissively licensed source codecited by 359
  4. 2024AutoCodeRover: Autonomous Program Improvementcited by 278
  5. 2024Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Modelscited by 122
  6. 2023Impact of Code Language Models on Automated Program Repaircited by 85
  7. 2022Less Training, More Repairing Please: Revisiting Automated Program Repair via Zero-shot Learningcited by 76
  8. 2022Automated Repair of Programs from Large Language Modelscited by 71
  9. 2023A Survey of Learning-based Automated Program Repaircited by 55
  10. 2023RepairLLaMA: Efficient Representations and Fine-Tuned Adapters for Program Repaircited by 50
  11. 2024Long Code Arena: a Set of Benchmarks for Long-Context Code Modelscited by 50
  12. 2023xCodeEval: A Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrievalcited by 47
  13. 2024A Systematic Literature Review on Large Language Models for Automated Program Repaircited by 37
  14. 2024On Leakage of Code Generation Evaluation Datasetscited by 32
  15. 2024CigaR: Cost-efficient Program Repair with LLMscited by 15
  16. 2024GitBug-Java: A Reproducible Benchmark of Recent Java Bugscited by 14
  17. 2019Bears: An Extensible Java Bug Benchmark for Automatic Program Repair Studiescited by 13
  18. 2024The Fact Selection Problem in LLM-Based Program Repaircited by 12
  19. 2023GitBug-Actions: Building Reproducible Bug-Fix Benchmarks with GitHub Actionscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 3.

9 papers.

  1. 2025debug-gym: A Text-Based Environment for Interactive Debuggingcited by 19
  2. 2025Benchmarking and Studying the LLM-based Agent System in End-to-End Software Developmentcited by 5
  3. 2025Benchmarking AI Models in Software Engineering: A Review, Search Tool, and Unified Approach for Elevating Benchmark Qualitycited by 4
  4. 2025Gradient-Based Program Repair: Fixing Bugs in Continuous Program Spacescited by 4
  5. 2026OmniCode: A Benchmark for Evaluating Software Engineering Agentscited by 4
  6. 2025Beyond Accuracy: Behavioral Dynamics of Agentic Multi-Hunk Repaircited by 2
  7. 2025AI-Assisted Fixes to Code Review Comments at Scalecited by 0
  8. 2026Fine-grained Approaches for Confidence Calibration of LLMs in Automated Code Revisioncited by 0
  9. 2026AuditRepairBench: A Paired-Execution Trace Corpus for Evaluator-Channel Ranking Instability in Agent Repaircited by 0