mlsanityData to 7 Oct 2026

Large Language Models and Simple, Stupid Bugs

arXiv 2303.11455 ↗First posted 20 Mar 2023cs.SE cs.CL cs.LGcited by 45

Not on the map. Why?

Abstract (arXiv)

With the advent of powerful neural language models, AI-based systems to assist developers in coding tasks are becoming widely available; Copilot is one such system. Copilot uses Codex, a large language model (LLM), to complete code conditioned on a preceding "prompt". Codex, however, is trained on public GitHub repositories, viz., on code that may include bugs and vulnerabilities. Previous studies [1], [2] show Codex reproduces vulnerabilities seen in training. In this study, we examine how prone Codex is to generate an interesting bug category, single statement bugs, commonly referred to as simple, stupid bugs or SStuBs in the MSR community. We find that Codex and similar LLMs do help avoid some SStuBs, but do produce known, verbatim SStuBs as much as 2x as likely than known, verbatim correct code. We explore the consequences of the Codex generated SStuBs and propose avoidance strategies that suggest the possibility of reducing the production of known, verbatim SStubs, and increase the possibility of producing known, verbatim fixes.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). Builds on 39 of its 64 references that are in the map. First posted on arXiv 20 Mar 2023.

Builds on

39 of 64 references.

References · 39

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  4. 2021Program Synthesis with Large Language Modelscited by 3,509
  5. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  6. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  7. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
  8. 2020GraphCodeBERT: Pre-training Code Representations with Data Flowcited by 600
  9. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
  10. 2021Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributionscited by 417
  11. 2021Unified Pre-training for Program Understanding and Generationcited by 361
  12. 2022A Systematic Evaluation of Large Language Models of Codecited by 360
27 more
  1. 2020Unsupervised Translation of Programming Languagescited by 262
  2. 2022Do Users Write More Insecure Code with AI Assistants?cited by 186
  3. 2020How Context Affects Language Models' Factual Predictionscited by 181
  4. 2018SequenceR: Sequence-to-Sequence Learning for End-to-End Program Repaircited by 144
  5. 2022Few-shot training LLMs for project-specific code-summarizationcited by 128
  6. 2018The Adverse Effects of Code Duplication in Machine Learning Models of Codecited by 123
  7. 2021Examining Zero-Shot Vulnerability Repair with Large Language Modelscited by 105
  8. 2021Jigsaw: Large Language Models meet Program Synthesiscited by 103
  9. 2022Evaluating the Text-to-SQL Capabilities of Large Language Modelscited by 101
  10. 2021A Neural Network Solves, Explains, and Generates University Math Problems by Program Synthesis and Few-Shot Learning at Human Levelcited by 96
  11. 2022Is GitHub's Copilot as Bad as Humans at Introducing Vulnerabilities in Code?cited by 63
  12. 2021Applying CodeBERT for Automated Program Repair of Java Simple Bugscited by 44
  13. 2022No More Fine-Tuning? An Experimental Evaluation of Prompt Tuning in Code Intelligencecited by 41
  14. 2022What is it like to program with artificial intelligence?cited by 39
  15. 2023Recommending Root-Cause and Mitigation Steps for Cloud Incidents using Large Language Modelscited by 38
  16. 2019How Often Do Single-Statement Bugs Occur? The ManySStuBs4J Datasetcited by 35
  17. 2021CodeTrans: Towards Cracking the Language of Silicon's Code Through Self-Supervised Deep Learning and High Performance Computingcited by 28
  18. 2021Automatic Program Repair with OpenAI's Codex: Evaluating QuixBugscited by 24
  19. 2021Choose Your Programming Copilot: A Comparison of the Program Synthesis Performance of GitHub Copilot and Genetic Programmingcited by 24
  20. 2022TSSB-3M: Mining single statement bugs at massive scalecited by 14
  21. 2022Codex Hacks HackerRank: Memorization Issues and a Framework for Code Synthesis Evaluationcited by 14
  22. 2021Solving Probability and Statistics Problems by Program Synthesiscited by 9
  23. 2022CodexDB: Generating Code for Processing SQL Queries using GPT-3 Codexcited by 5
  24. 2021On the Rise and Fall of Simple Stupid Bugs: a Life-Cycle Analysis of SStuBscited by 2
  25. 2021Mea culpa: How developers fix their own simple bugs differently from other developerscited by 2
  26. 2021A large-scale study on human-cloned changes for automated program repaircited by 2
  27. 2021On the Distribution of "Simple Stupid Bugs" in Unit Test Files: An Exploratory Studycited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 21; 2025: 12; 2026: 9.

45 papers.

Sort:

By area:

  1. 2024Beyond Functional Correctness: Exploring Hallucinations in LLM-Generated Codecited by 106
  2. 2023Can ChatGPT replace StackOverflow? A Study on Robustness and Reliability of Large Language Model Code Generationcited by 72
  3. 2024Bugs in Large Language Models Generated Code: An Empirical Studycited by 59
  4. 2024Calibration and Correctness of Language Models for Codecited by 49
  5. 2024A Performance Study of LLM-Generated Code on Leetcodecited by 48
  6. 2024Securing Large Language Models: Threats, Vulnerabilities and Responsible Practicescited by 36
  7. 2024Automatic Programming: Large Language Models and Beyondcited by 35
  8. 2024Prompting Techniques for Secure Code Generation: A Systematic Investigationcited by 33
  9. 2024AlphaVerus: Bootstrapping Formally Verified Code Generation through Self-Improving Translation and Treefinementcited by 33
  10. 2024CodeHalu: Investigating Code Hallucinations in LLMs via Execution-based Verificationcited by 32
  11. 2025Correctness Assessment of Code Generated by Large Language Models Using Internal Representationscited by 17
  12. 2024LLM-Powered Code Vulnerability Repair with Reinforcement Learning and Semantic Rewardcited by 16
33 more
  1. 2024Calibration of Large Language Models on Code Summarizationcited by 12
  2. 2024Studying LLM Performance on Closed- and Open-source Datacited by 11
  3. 2024Code Security Vulnerability Repair Using Reinforcement Learning with Large Language Modelscited by 10
  4. 2026Goedel-Code-Prover: Hierarchical Proof Search for Open State-of-the-Art Code Verificationcited by 10
  5. 2024A Deep Dive Into Large Language Model Code Generation Mistakes: What and Why?cited by 9
  6. 2025Hallucination by Code Generation LLMs: Taxonomy, Benchmarks, Mitigation, and Challengescited by 9
  7. 2025Risk Assessment Framework for Code LLMs via Leveraging Internal Statescited by 8
  8. 2023A Language Model of Java Methods with Train/Test Deduplicationcited by 6
  9. 2024Impeding LLM-assisted Cheating in Introductory Programming Assignments via Adversarial Perturbationcited by 5
  10. 2024AIGCodeSet: A New Annotated Dataset for AI Generated Code Detectioncited by 5
  11. 2024Enhancing Source Code Security with LLMs: Demystifying The Challenges and Generating Reliable Repairscited by 4
  12. 2024Harnessing the Power of LLMs: Automating Unit Test Generation for High-Performance Computingcited by 3
  13. 2025Where's the Bug? Attention Probing for Scalable Fault Localizationcited by 3
  14. 2025Localized Calibrated Uncertainty in Code Language Modelscited by 3
  15. 2026What Breaks When LLMs Code? Characterizing Operational Safety Failures of Agentic Code Assistantscited by 3
  16. 2024N-Version Assessment and Enhancement of Generative AIcited by 2
  17. 2025MigGPT: Harnessing Large Language Models for Automated Migration of Out-of-Tree Linux Kernel Patches Across Versionscited by 2
  18. 2025MultiAIGCD: A Comprehensive dataset for AI Generated Code Detection Covering Multiple Languages, Models,Prompts, and Scenarioscited by 2
  19. 2025Multicalibration for LLM-based Code Generationcited by 2
  20. 2024Morescient GAI for Software Engineering (Extended Version)cited by 1
  21. 2024Software Model Evolution with Large Language Models: Experiments on Simulated, Public, and Industrial Datasetscited by 1
  22. 2025Investigating Security Implications of Automatically Generated Code on the Software Supply Chaincited by 1
  23. 2025Open the Oyster: Empirical Evaluation and Improvement of Code Reasoning Confidence in LLMscited by 1
  24. 2025Anka: A Domain-Specific Language for Reliable LLM Code Generationcited by 1
  25. 2026Model See, Model Do? Exposure-Aware Evaluation of Bug-vs-Fix Preference in Code LLMscited by 1
  26. 2026From LLMs to Agents in Programming: The Impact of Providing an LLM with a Compilercited by 1
  27. 2026SPELL: Synthesis of Programmatic Edits using LLMscited by 1
  28. 2026Bridging Generation and Training: A Systematic Review of Quality Issues in LLMs for Codecited by 1
  29. 2026FLARE: Fine-Grained Diagnostic Feedback for LLM Code Refinementcited by 1
  30. 2023Breaking the Silence: the Threats of Using LLMs in Software Engineeringcited by 0
  31. 2025Evaluating the Limitations of Local LLMs in Solving Complex Programming Challengescited by 0
  32. 2026Where Did the Variability Go? From Vibe Coding to Product Lines by Regenerationcited by 0
  33. 2026What is the Difference Between Me and You? Benchmarking the Quality Gap Between Human-Written and AI-Generated Codecited by 0