mlsanityData to 14 Oct 2026

LLM-based Agents Suffer from Hallucinations: A Survey of Taxonomy, Methods, and Directions

arXiv 2509.18970 ↗First posted 23 Sep 2025cs.AIcited by 50

On the map: LLM agents and tool use → Medical and scientific domain agents

Abstract (arXiv)

Driven by the rapid advancements of Large Language Models (LLMs), LLM-based agents have emerged as powerful intelligent systems capable of human-like cognition, reasoning, and interaction. These agents are increasingly being deployed across diverse real-world applications, including student education, scientific research, and financial analysis. However, despite their remarkable potential, LLM-based agents remain vulnerable to hallucination issues, which can result in erroneous task execution and undermine the reliability of the overall system design. Addressing this critical challenge requires a deep understanding and a systematic consolidation of recent advances on LLM-based agents. To this end, we present the first comprehensive survey of hallucinations in LLM-based agents. By carefully analyzing the complete workflow of agents, we propose a new taxonomy that identifies different types of agent hallucinations occurring at different stages. Furthermore, we conduct an in-depth examination of eighteen triggering causes underlying the emergence of agent hallucinations. Through a detailed review of a large number of existing studies, we summarize approaches for hallucination mitigation and detection, and highlight promising directions for future research. We hope this survey will inspire further efforts toward addressing hallucinations in LLM-based agents, ultimately contributing to the development of more robust and reliable agent systems.

Key facts. Cited by 50 papers in the map (data through 14 Oct 2026). Placed in Medical and scientific domain agents, one of 16 directions in LLM agents and tool use. Builds on 234 of its 324 references that are in the map. First posted on arXiv 23 Sep 2025.

Builds on

234 of 324 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025SEVADE: Self-Evolving Multi-Agent Analysis with Decoupled Evaluation for Hallucination-Resistant Irony Detection9 Aug
  2. 2025EH-Benchmark Ophthalmic Hallucination Benchmark and Agent-Driven Top-Down Traceable Reasoning Workflow24 Jul
  3. 2025MedAgent-Pro: Towards Evidence-based Multi-modal Medical Diagnosis via Reasoning Agentic Workflow21 Mar
  4. 2025Hallucination Mitigation using Agentic AI Natural Language-Based Frameworks19 Jan
  5. 2024Small Agent Can Also Rock! Empowering Small Language Models as Hallucination Detector17 Jun
  6. 2024Towards Detecting LLMs Hallucination via Markov Chain-based Multi-agent Debate Framework5 Jun
  7. 2023Conversational Health Agents: A Personalized LLM-Powered Agent Framework3 Oct

Other references · 227

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  8. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  9. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  10. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  11. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  12. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
38 more
  1. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,395
  2. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
  3. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  4. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
  5. 2022Editing Models with Task Arithmeticcited by 1,120
  6. 2025A-MEM: Agentic Memory for LLM Agentscited by 1,063
  7. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 1,011
  8. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
  9. 2024Evaluating Very Long-Term Conversational Memory of LLM Agentscited by 830
  10. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 783
  11. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 742
  12. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  13. 2022A Survey on In-context Learningcited by 721
  14. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  15. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 688
  16. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 670
  17. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 634
  18. 2025Why Do Multi-Agent LLM Systems Fail?cited by 604
  19. 2022Towards Reasoning in Large Language Models: A Surveycited by 596
  20. 2021Editing Factual Knowledge in Language Modelscited by 553
  21. 2025Visual-RFT: Visual Reinforcement Fine-Tuningcited by 513
  22. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 509
  23. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 478
  24. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 445
  25. 2025Multi-Agent Collaboration Mechanisms: A Survey of LLMscited by 431
  26. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 412
  27. 2025Zep: A Temporal Knowledge Graph Architecture for Agent Memorycited by 400
  28. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Casescited by 393
  29. 2025ReTool: Reinforcement Learning for Strategic Tool Use in LLMscited by 392
  30. 2023Who's Harry Potter? Approximate Unlearning in LLMscited by 381
  31. 2023Editing Large Language Models: Problems, Methods, and Opportunitiescited by 372
  32. 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 341
  33. 2020Keystroke Dynamics as Part of Lifeloggingcited by 334
  34. 2023Reasoning on Graphs: Faithful and Interpretable Large Language Model Reasoningcited by 328
  35. 2023Trusting Your Evidence: Hallucinate Less with Context-aware Decodingcited by 327
  36. 2024Agent-SafetyBench: Evaluating the Safety of LLM Agentscited by 306
  37. 2023A Dynamic LLM-Powered Agent Network for Task-Oriented Agent Collaborationcited by 290
  38. 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agentscited by 288

50 most cited of 227 All 227 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 43.

50 papers.

Sort:

By area:

  1. 2026From Agent Traces to Trust: A Survey of Evidence Tracing and Execution Provenance in LLM Agentscited by 37
  2. 2026HarnessCompass: Guiding Automatic Harness Evolution toward Generalizable and Effective Agent Harnessescited by 17
  3. 2026SafeHarness: Lifecycle-Integrated Security Architecture for LLM-based Agent Deploymentcited by 15
  4. 2026Characterizing Faults in Agentic AI: A Taxonomy of Types, Symptoms, and Root Causescited by 12
  5. 2025Sherlock: Reliable and Efficient Agentic Workflow Executioncited by 11
  6. 2025SoK: Trust-Authorization Mismatch in LLM Agent Interactionscited by 11
  7. 2025Automatically Benchmarking LLM Code Agents through Agent-Driven Annotation and Evaluationcited by 9
  8. 2026TerraScope: Pixel-Grounded Visual Reasoning for Earth Observationcited by 7
  9. 2026From Chatbot to Digital Colleague: The Paradigm Shift Toward Persistent Autonomous AIcited by 6
  10. 2026Why Your Deep Research Agent Fails? On Hallucination Evaluation in Full Research Trajectorycited by 5
  11. 2026Complementary RL: Towards Efficient Experience-Driven Agent Learningcited by 5
  12. 2026Real-Time Detection and Repair of LLM Agent Failurescited by 5
38 more
  1. 2025MAD-Fact: A Multi-Agent Debate Framework for Long-Form Factuality Evaluation in LLMscited by 4
  2. 2026Reasoning over Precedents Alongside Statutes: Case-Augmented Deliberative Alignment for LLM Safetycited by 3
  3. 2026ProAct: Agentic Lookahead in Interactive Environmentscited by 3
  4. 2026TrustTrade: Human-Inspired Selective Consensus Reduces Decision Uncertainty in LLM Trading Agentscited by 3
  5. 2026Safety, Security, and Cognitive Risks in World Modelscited by 3
  6. 2026Do LLMs Know Tool Irrelevance? Demystifying Structural Alignment Bias in Tool Invocationscited by 3
  7. 2026CIA: Inferring the Communication Topology from LLM-based Multi-Agent Systemscited by 3
  8. 2026AgentCollabBench: Diagnosing When Good Agents Make Bad Collaboratorscited by 3
  9. 2025Mitigating Hallucination in Large Language Models (LLMs): An Application-Oriented Survey on RAG, Reasoning, and Agentic Systemscited by 2
  10. 2025The Evolution of Agentic AI in Cybersecurity: From Single LLM Reasoners to Multi-Agent Systems and Autonomous Pipelinescited by 2
  11. 2026Emergence of Fragility in LLM-based Social Networks: the Case of Moltbookcited by 2
  12. 2026Program Analysis Guided LLM Agent for Proof-of-Concept Generationcited by 2
  13. 2026Small Model as Master Orchestrator: Learning Unified Agent-Tool Orchestration with Parallel Subtask Decompositioncited by 2
  14. 2026Hallucination as Context Drift: Synchronization Protocols for Multi-Agent LLM Systemscited by 2
  15. 2026Online Monitoring and Corrective Steering of Programming Agentscited by 2
  16. 2026The Shadow Self: Intrinsic Value Misalignment in Large Language Model Agentscited by 1
  17. 2026Differentiable Modal Logic for Multi-Agent Diagnosis, Orchestration and Communicationcited by 1
  18. 2026CODE-GEN: A Human-in-the-Loop RAG-Based Agentic AI System for Multiple-Choice Question Generationcited by 1
  19. 2026Plausible but Wrong: A case study on Agentic Failures in Astrophysical Workflowscited by 1
  20. 2026Taming "Zombie'' Agents: A Markov State-Aware Framework for Resilient Multi-Agent Evolutioncited by 1
  21. 2026Lean4Agent: Formal Modeling and Verification for Agent Workflow and Trajectorycited by 1
  22. 2026Linguistic Firewall: Geometry as Defense in Multi-Agent Systems Routingcited by 1
  23. 2026FinPersona-Bench: A Benchmark for Longitudinal Psychometric Stability of Autonomous Financial Agentscited by 1
  24. 2026BrainPilot: Automating Brain Discovery with Agentic Researchcited by 1
  25. 2025GrowthHacker: Automated Off-Policy Evaluation Optimization Using Code-Modifying LLM Agentscited by 0
  26. 2026MuVaC: A Variational Causal Framework for Multimodal Sarcasm Understanding in Dialoguescited by 0
  27. 2026On the Suitability of LLM-Driven Agents for Dark Pattern Auditscited by 0
  28. 2026EA-Agent: A Structured Multi-Step Reasoning Agent for Entity Alignmentcited by 0
  29. 2026AgriIR: A Scalable Framework for Domain-Specific Knowledge Retrievalcited by 0
  30. 2026Reliable AI Needs to Externalize Implicit Knowledge: A Human-AI Collaboration Perspectivecited by 0
  31. 2026DeepRoot: A KG-Coordinated Multi-Agent System for Therapeutic Reasoning over Historical Medical Textscited by 0
  32. 2026Swarm-Driven Multi-Agent Reasoning for Smart City Securitycited by 0
  33. 2026Beware of Agentic Botnets: Scalable Untargeted Promptware Attacks via Universal and Transferable Adversarial HalluSquattingcited by 0
  34. 2026Protective Capacity Hallucination: When Large Language Models Claim Nonexistent Capabilitiescited by 0
  35. 2026An Agentic Interface for End-to-End Probabilistic Seismic Hazard and Risk Analysiscited by 0
  36. 2026LLMs and Agentic AI Systems for Smart Grids: A Tutorial on Architectures and Applicationscited by 0
  37. 2026LexAgentHallu: A Hierarchical Benchmark for Profiling Hallucinations in Legal Agentscited by 0
  38. 2026MIRAGE: How Conversation State Shapes Historical Evidence Use in Multimodal Personal Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026AgentHallu: Benchmarking Automated Hallucination Attribution of LLM-based Agents
  2. 2025MIRAGE-Bench: LLM Agent is Hallucinating and Where to Find Them
  3. 2025HEAL: An Empirical Study on Hallucinations in Embodied Agents Driven by Large Language Models
  4. 2024Investigating the Role of Prompting and External Tools in Hallucination Rates of Large Language Models
  5. 2025From "Hallucination" to "Suture": Insights from Language Philosophy to Enhance Large Language Models
  6. 2024Good Parenting is all you need -- Multi-agentic LLM Hallucination Mitigation
  7. 2025Towards Mitigation of Hallucination for LLM-empowered Agents: Progressive Generalization Bound Exploration and Watchdog Monitor
  8. 2026PRISM: Probing Reasoning, Instruction, and Source Memory in LLM Hallucinations