mlsanityData to 14 Oct 2026

Hallucination by Code Generation LLMs: Taxonomy, Benchmarks, Mitigation, and Challenges

arXiv 2504.20799 ↗First posted 29 Apr 2025cs.SE cs.AIcited by 11

Not on the map. Why?

Abstract (arXiv)

Recent technical breakthroughs in large language models (LLMs) have enabled them to fluently generate source code. Software developers often leverage both general-purpose and code-specialized LLMs to revise existing code or even generate a whole function from scratch. These capabilities are also beneficial in no-code or low-code contexts, in which one can write programs without a technical background. However, due to their internal design, LLMs are prone to generating hallucinations, which are incorrect, nonsensical, and not justifiable information but difficult to identify its presence. This problem also occurs when generating source code. Once hallucinated code is produced, it is often challenging for users to identify and fix it, especially when such hallucinations can be identified under specific execution paths. As a result, the hallucinated code may remain unnoticed within the codebase. This survey investigates recent studies and techniques relevant to hallucinations generated by CodeLLMs. We categorize the types of hallucinations in the code generated by CodeLLMs, review existing benchmarks and mitigation strategies, and identify open challenges. Based on these findings, this survey outlines further research directions in the detection and removal of hallucinations produced by CodeLLMs.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 40 of its 62 references that are in the map. First posted on arXiv 29 Apr 2025.

Builds on

40 of 62 references.

References · 40

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  2. 2021Program Synthesis with Large Language Modelscited by 3,637
  3. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  4. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  5. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  6. 2021Measuring Coding Challenge Competence With APPScited by 902
  7. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 885
  8. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 744
  9. 2024A Survey on Large Language Models for Code Generationcited by 658
  10. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 452
  11. 2021Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributionscited by 431
  12. 2021PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Modelscited by 411
28 more
  1. 2023CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-Xcited by 300
  2. 2023LEVER: Learning to Verify Language-to-Code Generation with Executioncited by 239
  3. 2023ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generationcited by 168
  4. 2023TACO: Topics in Algorithmic COde generation datasetcited by 160
  5. 2024Beyond Functional Correctness: Exploring Hallucinations in LLM-Generated Codecited by 107
  6. 2022Repository-Level Prompt Generation for Large Language Models of Codecited by 106
  7. 2024LLM Hallucinations in Practical Code Generation: Phenomena, Mechanism, and Mitigationcited by 93
  8. 2023How Secure is Code Generated by ChatGPT?cited by 81
  9. 2023Refining ChatGPT-Generated Code: Characterizing and Mitigating Code Quality Issuescited by 79
  10. 2023Can ChatGPT replace StackOverflow? A Study on Robustness and Reliability of Large Language Model Code Generationcited by 73
  11. 2024EvoCodeBench: An Evolving Code Generation Benchmark Aligned with Real-World Code Repositoriescited by 72
  12. 2024We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMscited by 65
  13. 2023Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Codecited by 62
  14. 2024Bugs in Large Language Models Generated Code: An Empirical Studycited by 60
  15. 2024What's Wrong with Your Code Generated by Large Language Models? An Extensive Studycited by 53
  16. 2024Before Generation, Align it! A Novel and Effective Strategy for Mitigating Hallucinations in Text-to-SQL Generationcited by 50
  17. 2023Large Language Models and Simple, Stupid Bugscited by 45
  18. 2024RLCoder: Reinforcement Learning for Repository-Level Code Completioncited by 44
  19. 2024MMCode: Benchmarking Multimodal Large Language Models for Code Generation with Visually Rich Programming Problemscited by 40
  20. 2023SALLM: Security Assessment of Generated Codecited by 39
  21. 2023CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generationcited by 39
  22. 2023ToolCoder: Teach Code Generation Models to use API search toolscited by 37
  23. 2024CodeHalu: Investigating Code Hallucinations in LLMs via Execution-based Verificationcited by 34
  24. 2024Teaching Code LLMs to Use Autocompletion Tools in Repository-Level Code Generationcited by 30
  25. 2024On Mitigating Code LLM Hallucinations with API Documentationcited by 21
  26. 2024De-Hallucinator: Mitigating LLM Hallucinations in Code Generation Tasks via Iterative Groundingcited by 15
  27. 2024Collu-Bench: A Benchmark for Predicting Language Model Hallucinations in Codecited by 13
  28. 2021Multi-modal Program Inference: a Marriage of Pre-trainedLanguage Models and Component-based Synthesiscited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 8.

11 papers.

  1. 2026Delulu: A Verified Multi-Lingual Benchmark for Code Hallucination Detection in Fill-in-the-Middle Taskscited by 3
  2. 2025Hallucination in LLM-Based Code Generation: An Automotive Case Studycited by 2
  3. 2025Library Hallucinations in LLM-Generated Code: A Risk Analysis Grounded in Developer Queriescited by 1
  4. 2025CodeFuse-CommitEval: Towards Benchmarking LLM's Power on Commit Message and Code Change Inconsistency Detectioncited by 1
  5. 2026Detecting and Correcting Hallucinations in LLM-Generated Code via Deterministic AST Analysiscited by 1
  6. 2026SOCIA-EVO: Automated Simulator Construction via Dual-Anchored Bi-Level Optimizationcited by 0
  7. 2026Task Abstention for Large Language Models in Code Generationcited by 0
  8. 2026A Tertiary Review of Large Language Model-Based Code Generating Tasks: Trends, Challenges, and Future Directionscited by 0
  9. 2026Evaluating Inference-Time Defenses Against Package Hallucination in LLM-Generated Codecited by 0
  10. 2026Better Understanding, Better Fixes? A Study of Hallucination in LLM-based Automated Program Repaircited by 0
  11. 2026Hallucination Across the Reasoning Lifecycle: Interface Visibility, Causal Evidence, and Release Control in Large Reasoning Modelscited by 0