mlsanityData to 7 Oct 2026

Large Language Models of Code Fail at Completing Code with Potential Bugs

arXiv 2306.03438 ↗First posted 6 Jun 2023cs.LG cs.AI cs.CLcited by 28

Not on the map. Why?

Abstract (arXiv)

Large language models of code (Code-LLMs) have recently brought tremendous advances to code completion, a fundamental feature of programming assistance and code intelligence. However, most existing works ignore the possible presence of bugs in the code context for generation, which are inevitable in software development. Therefore, we introduce and study the buggy-code completion problem, inspired by the realistic scenario of real-time code suggestion where the code context contains potential bugs -- anti-patterns that can become bugs in the completed program. To systematically study the task, we introduce two datasets: one with synthetic bugs derived from semantics-altering operator changes (buggy-HumanEval) and one with realistic bugs derived from user submissions to coding problems (buggy-FixEval). We find that the presence of potential bugs significantly degrades the generation performance of the high-performing Code-LLMs. For instance, the passing rates of CODEGEN-2B-MONO on test cases of buggy-HumanEval drop more than 50% given a single potential bug in the context. Finally, we investigate several post-hoc methods for mitigating the adverse effect of potential bugs and find that there remains a significant gap in post-mitigation performance.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Builds on 43 of its 75 references that are in the map. First posted on arXiv 6 Jun 2023.

Builds on

43 of 75 references.

References · 43

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  2. 2021Program Synthesis with Large Language Modelscited by 3,509
  3. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  4. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
  5. 2021Measuring Coding Challenge Competence With APPScited by 879
  6. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  7. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
  8. 2020CodeBLEU: a Method for Automatic Evaluation of Code Synthesiscited by 468
  9. 2018Threat of Adversarial Attacks on Deep Learning in Computer Vision: A Surveycited by 459
  10. 2021Unified Pre-training for Program Understanding and Generationcited by 361
  11. 2020IntelliCode Compose: Code Generation Using Transformercited by 225
  12. 2017A Survey of Machine Learning for Big Code and Naturalnesscited by 220
31 more
  1. 2017Learning to Represent Programs with Graphscited by 219
  2. 2021CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Taskscited by 199
  3. 2018SequenceR: Sequence-to-Sequence Learning for End-to-End Program Repaircited by 144
  4. 2022ReACC: A Retrieval-Augmented Code Completion Frameworkcited by 117
  5. 2018An Empirical Study on Learning Bug-Fixing Patches in the Wild via Neural Machine Translationcited by 106
  6. 2022Productivity Assessment of Neural Code Completioncited by 97
  7. 2020Graph-based, Self-Supervised Program Repair from Diagnostic Feedbackcited by 93
  8. 2020PyMT5: multi-mode translation of natural language and Python code with transformerscited by 80
  9. 2020Opportunities and Challenges in Deep Learning Adversarial Robustness: A Surveycited by 71
  10. 2021Break-It-Fix-It: Unsupervised Learning for Program Repaircited by 71
  11. 2020Code Prediction by Feeding Trees to Transformerscited by 70
  12. 2018Generative Code Modeling with Graphscited by 59
  13. 2021Self-Supervised Bug Detection and Repaircited by 59
  14. 2020Multi-task Learning based Pre-trained Language Model for Code Completioncited by 58
  15. 2020Deep Learning for Source Code Modeling and Generation: Models, Applications and Challengescited by 55
  16. 2017Code Completion with Neural Attention and Pointer Networkscited by 49
  17. 2020Semantic Robustness of Models of Source Codecited by 43
  18. 2019Neural Program Repair by Jointly Learning to Localize and Repaircited by 39
  19. 2019How Often Do Single-Statement Bugs Occur? The ManySStuBs4J Datasetcited by 35
  20. 2019Learning and Evaluating Contextual Embedding of Source Codecited by 33
  21. 2020Backdoors in Neural Models of Source Codecited by 27
  22. 2021Learning to Complete Code with Sketchescited by 23
  23. 2014Mining Idioms from Source Codecited by 19
  24. 2020Learning Autocompletion from Real-World Datasetscited by 19
  25. 2022On Distribution Shift in Learning-based Bug Detectorscited by 15
  26. 2023Better Context Makes Better Code Language Models: A Case Study on Function Call Argument Completioncited by 14
  27. 2018Learning How to Mutate Source Code from Bug-Fixescited by 13
  28. 2020Review4Repair: Code Review Aided Automatic Program Repairingcited by 13
  29. 2022HEAT: Hyperedge Attention Networkscited by 13
  30. 2022Can we learn from developer mistakes? Learning to localize and repair real bugs from real bug fixescited by 4
  31. 2021Learning to Extend Program Graphs to Work-in-Progress Codecited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 9; 2025: 8; 2026: 8.

28 papers.

Sort:

By area:

  1. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  2. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 389
  3. 2024The RealHumanEval: Evaluating Large Language Models' Abilities to Support Programmerscited by 28
  4. 2024CodeMirage: Hallucinations in Code Generated by Large Language Modelscited by 23
  5. 2024IRCoder: Intermediate Representations Make Language Models Robust Multilingual Code Generatorscited by 21
  6. 2024The Counterfeit Conundrum: Can Code Language Models Grasp the Nuances of Their Incorrect Generations?cited by 20
  7. 2026Large Language Model Reasoning Failurescited by 20
  8. 2025Improving Chain-of-Thought Reasoning via Quasi-Symbolic Abstractionscited by 17
  9. 2023Machine-assisted quantitizing designs: augmenting humanities and social sciences with artificial intelligencecited by 13
  10. 2024LLMs with Industrial Lens: Deciphering the Challenges and Prospects -- A Surveycited by 12
  11. 2024FVEval: Understanding Language Model Capabilities in Formal Verification of Digital Hardwarecited by 9
  12. 2025Intuition to Evidence: Measuring AI's True Impact on Developer Productivitycited by 5
16 more
  1. 2024Deploying Foundation Model Powered Agent Services: A Surveycited by 4
  2. 2024No Size Fits All: The Perils and Pitfalls of Leveraging LLMs Vary with Company Sizecited by 3
  3. 2025PerfCoder: Large Language Models for Interpretable Code Performance Optimizationcited by 3
  4. 2026Falsification, Not Exposure: An Internally Preregistered Placebo-Controlled Decomposition of Self-Repair Feedback in Frozen Small Code Modelscited by 3
  5. 2024Envisioning the Next-Generation AI Coding Assistants: Insights & Proposalscited by 2
  6. 2025SOK: Exploring Hallucinations and Security Risks in AI-Assisted Software Development with Insights for LLM Deploymentcited by 2
  7. 2025Program Semantic Inequivalence Game with Large Language Modelscited by 1
  8. 2026VeriInteresting: An Empirical Study of Model Prompt Interactions in Verilog Code Generationcited by 1
  9. 2026Belief-Guided Inference Control for Large Language Model Services via Verifiable Observationscited by 1
  10. 2026ARIADNE: Agentic Reward-Informed Adaptive Decision Exploration via Blackboard-Driven MCTS for Competitive Program Generationcited by 1
  11. 2026On the Robustness of LLMs' Internal Representation of Code Correctnesscited by 1
  12. 2025LLM Based Long Code Translation using Identifier Replacementcited by 0
  13. 2025Learning Bug Context for PyTorch-to-JAX Translation with LLMscited by 0
  14. 2025DebugTA: An LLM-Based Agent for Simplifying Debugging and Teaching in Programming Educationcited by 0
  15. 2026Tricky2: Towards a Benchmark for Evaluating Human and LLM Error Interactionscited by 0
  16. 2026On the Lexical Superstition of Large Language Models for Code Comprehension: Re-evaluation on Code of Low Lexical Qualitycited by 0