mlsanityData to 7 Oct 2026

LLM Hallucinations in Practical Code Generation: Phenomena, Mechanism, and Mitigation

arXiv 2409.20550 ↗First posted 30 Sep 2024cs.SE cs.AI cs.CLcited by 92

Not on the map. Why?

Cited by 28 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Code generation aims to automatically generate code from input requirements, significantly enhancing development efficiency. Recent large language models (LLMs) based approaches have shown promising results and revolutionized code generation task. Despite the promising performance, LLMs often generate contents with hallucinations, especially for the code generation scenario requiring the handling of complex contextual dependencies in practical development process. Although previous study has analyzed hallucinations in LLM-powered code generation, the study is limited to standalone function generation. In this paper, we conduct an empirical study to study the phenomena, mechanism, and mitigation of LLM hallucinations within more practical and complex development contexts in repository-level generation scenario. First, we manually examine the code generation results from six mainstream LLMs to establish a hallucination taxonomy of LLM-generated code. Next, we elaborate on the phenomenon of hallucinations, analyze their distribution across different models. We then analyze causes of hallucinations and identify four potential factors contributing to hallucinations. Finally, we propose an RAG-based mitigation method, which demonstrates consistent effectiveness in all studied LLMs. The replication package including code, data, and experimental results is available at https://github.com/DeepSoftwareAnalytics/LLMCodingHallucination

Key facts. Cited by 92 papers in the map (data through 7 Oct 2026). Builds on 62 of its 81 references that are in the map. First posted on arXiv 30 Sep 2024.

Builds on

62 of 81 references.

References · 62

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  6. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  7. 2021Program Synthesis with Large Language Modelscited by 3,509
  8. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  9. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  10. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  11. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  12. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
38 more
  1. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  2. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  3. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
  4. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  5. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  6. 2023StarCoder: may the source be with you!cited by 790
  7. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 787
  8. 2024Large Language Model based Multi-Agents: A Survey of Progress and Challengescited by 752
  9. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  10. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  11. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  12. 2020GraphCodeBERT: Pre-training Code Representations with Data Flowcited by 600
  13. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527
  14. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
  15. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 484
  16. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
  17. 2023RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generationcited by 357
  18. 2022UniXcoder: Unified Cross-Modal Pre-training for Code Representationcited by 355
  19. 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 327
  20. 2024A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Modelscited by 266
  21. 2022Grounded Copilot: How Programmers Interact with Code-Generating Modelscited by 216
  22. 2023ART: Automatic multi-step reasoning and tool-use for large language modelscited by 193
  23. 2023SantaCoder: don't reach for the stars!cited by 169
  24. 2023ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generationcited by 166
  25. 2022GitHub Copilot AI pair programmer: Asset or Liability?cited by 150
  26. 2021PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computationcited by 134
  27. 2023Do Language Models Know When They're Hallucinating References?cited by 115
  28. 2024Beyond Functional Correctness: Exploring Hallucinations in LLM-Generated Codecited by 106
  29. 2023Cognitive Mirage: A Review of Hallucinations in Large Language Modelscited by 83
  30. 2023Can ChatGPT replace StackOverflow? A Study on Robustness and Reliability of Large Language Model Code Generationcited by 72
  31. 2024EvoCodeBench: An Evolving Code Generation Benchmark Aligned with Real-World Code Repositoriescited by 71
  32. 2024DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositoriescited by 71
  33. 2023Towards an Understanding of Large Language Models in Software Engineering Taskscited by 67
  34. 2023A Survey of Large Language Models for Code: Evolution, Benchmarking, and Future Trendscited by 66
  35. 2023Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Codecited by 62
  36. 2022When Language Model Meets Private Librarycited by 54
  37. 2022PanGu-Coder: Program Synthesis with Function-Level Language Modelingcited by 47
  38. 2024RLCoder: Reinforcement Learning for Repository-Level Code Completioncited by 44

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 31; 2026: 58.

92 papers.

Sort:

By area:

  1. 2026Large Language Model Reasoning Failurescited by 20
  2. 2026Uncovering Security Threats and Architecting Defenses in Autonomous Agents: A Case Study of OpenClawcited by 18
  3. 2025Correctness Assessment of Code Generated by Large Language Models Using Internal Representationscited by 17
  4. 2025An Empirical Study of Agent Developer Practices in AI Agent Frameworkscited by 14
  5. 2024Top General Performance = Top Domain Performance? DomainCodeBench: A Multi-domain Code Generation Benchmarkcited by 13
  6. 2025HiFo-Prompt: Prompting with Hindsight and Foresight for LLM-based Automatic Heuristic Designcited by 13
  7. 2025Blueprint First, Model Second: A Framework for Deterministic LLM Workflowcited by 11
  8. 2026Characterizing Faults in Agentic AI: A Taxonomy of Types, Symptoms, and Root Causescited by 11
  9. 2025ReSeek: A Self-Correcting Framework for Search Agents with Instructive Rewardscited by 10
  10. 2024A Deep Dive Into Large Language Model Code Generation Mistakes: What and Why?cited by 9
  11. 2025Hallucination by Code Generation LLMs: Taxonomy, Benchmarks, Mitigation, and Challengescited by 9
  12. 2025Towards Formal Verification of LLM-Generated Code from Natural Language Promptscited by 8
38 more
  1. 2026AlignCoder: Aligning Retrieval with Target Intent for Repository-Level Code Completioncited by 7
  2. 2025Towards Mitigating API Hallucination in Code Generated by LLMs with Hierarchical Dependency Awarecited by 6
  3. 2025Takedown: How It's Done in Modern Coding Agent Exploitscited by 6
  4. 2026RovoDev Code Reviewer: A Large-Scale Online Evaluation of LLM-based Code Review Automation at Atlassiancited by 6
  5. 2024STACKFEED: Structured Textual Actor-Critic Knowledge Base Editing with FeedBackcited by 4
  6. 2025HCRMP: A LLM-Hinted Contextual Reinforcement Learning Framework for Autonomous Drivingcited by 4
  7. 2025EyeMulator: Improving Code Language Models by Mimicking Human Visual Attentioncited by 4
  8. 2025Past, Present, and Future of Bug Tracking in the Generative AI Eracited by 4
  9. 2026Trust Me, Import This: Dependency Steering Attacks via Malicious Agent Skillscited by 4
  10. 2025Curiosity by Design: An LLM-based Coding Assistant Asking Clarification Questionscited by 3
  11. 2025Securing LLM-Generated Embedded Firmware through AI Agent-Driven Validation and Patchingcited by 3
  12. 2026DRAINCODE: Stealthy Energy Consumption Attacks on Retrieval-Augmented Code Generation via Context Poisoningcited by 3
  13. 2025SOK: Exploring Hallucinations and Security Risks in AI-Assisted Software Development with Insights for LLM Deploymentcited by 2
  14. 2025Isolating Language-Coding from Problem-Solving: Benchmarking LLMs with PseudoEvalcited by 2
  15. 2025Hallucination in LLM-Based Code Generation: An Automotive Case Studycited by 2
  16. 2025MOSAIC: Multi-agent Orchestration for Task-Intelligent Scientific Codingcited by 2
  17. 2025FELA: A Multi-Agent Evolutionary System for Feature Engineering of Industrial Event Log Datacited by 2
  18. 2025SLMFix: Leveraging Small Language Models for Domain Specific Language Error Fixing with Reinforcement Learningcited by 2
  19. 2026GAIA: A Data Flywheel System for Training GUI Test-Time Scaling Critic Modelscited by 2
  20. 2026ManiBench: A Benchmark for Testing Visual-Logic Drift and Syntactic Hallucinations in Manim Code Generationcited by 2
  21. 2026PairCoder++: Pair Programming as a Universal Paradigm for Verified Code-Driven Multimodal and Structured-Artifact Generationcited by 2
  22. 2026PhoenixRepair: Rethinking Repair Strategy Exploration in Software Agentscited by 2
  23. 2025Automated Factual Benchmarking for In-Car Conversational Systems using Large Language Modelscited by 1
  24. 2025An AI-Powered Research Assistant in the Lab: A Practical Guide for Text Analysis Through Iterative Collaboration with LLMscited by 1
  25. 2025SAFE: Improving LLM Systems using Sentence-Level In-generation Attributioncited by 1
  26. 2025Web3 x AI Agents: Landscape, Integrations, and Foundational Challengescited by 1
  27. 2025Investigating Security Implications of Automatically Generated Code on the Software Supply Chaincited by 1
  28. 2025LSPRAG: LSP-Guided RAG for Language-Agnostic Real-Time Unit Test Generationcited by 1
  29. 2025From LLMs to Agents: A Comparative Evaluation of LLMs and LLM-based Agents in Security Patch Detectioncited by 1
  30. 2025CodeFuse-CommitEval: Towards Benchmarking LLM's Power on Commit Message and Code Change Inconsistency Detectioncited by 1
  31. 2026SolAgent: A Specialized Multi-Agent Framework for Solidity Code Generationcited by 1
  32. 2026Unseen-Codebases-Domain Data Synthesis and Training Based on Code Graphscited by 1
  33. 2026PMAx: An Agentic Framework for AI-Driven Process Miningcited by 1
  34. 2026PIDP-Attack: Combining Prompt Injection with Database Poisoning Attacks on Retrieval-Augmented Generation Systemscited by 1
  35. 2026Automated near-term quantum algorithm discovery for molecular ground statescited by 1
  36. 2026Aligned Agents, Biased Swarm: Measuring Bias Amplification in Multi-Agent Systemscited by 1
  37. 2026MR-Coupler: Automated Metamorphic Test Generation via Functional Coupling Analysiscited by 1
  38. 2026ReFEree: Reference-Free and Fine-Grained Method for Evaluating Factual Consistency in Real-World Code Summarizationcited by 1

50 most cited of 92 All 92 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024De-Hallucinator: Mitigating LLM Hallucinations in Code Generation Tasks via Iterative Grounding
  2. 2026Evaluating Inference-Time Defenses Against Package Hallucination in LLM-Generated Code
  3. 2026Refusing the Impossible: A Taxonomy and Benchmark for Code Hallucination in Large Language Models ↗
  4. 2026Synthetic Hallucinations, Real Gains: Hard Negatives from Frontier Models for FIM Hallucination Mitigation
  5. 2024Understanding Defects in Generated Codes by Language Models
  6. 2025Towards Advancing Code Generation with Large Language Models: A Research Roadmap
  7. 2025Refining Critical Thinking in LLM Code Generation: A Faulty Premise-based Evaluation Framework
  8. 2026HalluJudge: A Reference-Free Hallucination Detection for Context Misalignment in Code Review Automation