mlsanityData to 14 Oct 2026

HEAL: An Empirical Study on Hallucinations in Embodied Agents Driven by Large Language Models

arXiv 2506.15065 ↗First posted 18 Jun 2025cs.LG cs.ROcited by 6

On the map: LLM agents and tool use → Medical and scientific domain agents

Abstract (arXiv)

Large language models (LLMs) are increasingly being adopted as the cognitive core of embodied agents. However, inherited hallucinations, which stem from failures to ground user instructions in the observed physical environment, can lead to navigation errors, such as searching for a refrigerator that does not exist. In this paper, we present the first systematic study of hallucinations in LLM-based embodied agents performing long-horizon tasks under scene-task inconsistencies. Our goal is to understand to what extent hallucinations occur, what types of inconsistencies trigger them, and how current models respond. To achieve these goals, we construct a hallucination probing set by building on an existing benchmark, capable of inducing hallucination rates up to 40x higher than base prompts. Evaluating 12 models across two simulation environments, we find that while models exhibit reasoning, they fail to resolve scene-task inconsistencies-highlighting fundamental limitations in handling infeasible tasks. We also provide actionable insights on ideal model behavior for each scenario, offering guidance for developing more robust and reliable planning strategies.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Medical and scientific domain agents, one of 16 directions in LLM agents and tool use. Builds on 28 of its 47 references that are in the map. First posted on arXiv 18 Jun 2025.

Builds on

28 of 47 references.

References · 28

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  2. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  3. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  4. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  5. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 819
  6. 2018Object Hallucination in Image Captioningcited by 669
  7. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 445
  8. 2018VirtualHome: Simulating Household Activities via Programscited by 415
  9. 2023A Survey of Hallucination in Large Foundation Modelscited by 368
  10. 2024Robotic Control via Embodied Chain-of-Thought Reasoningcited by 346
  11. 2023Robots That Ask For Help: Uncertainty Alignment for Large Language Model Plannerscited by 308
  12. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluationcited by 300
16 more
  1. 2023Fine-tuning Language Models for Factualitycited by 221
  2. 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agentscited by 221
  3. 2024AutoDAN-Turbo: A Lifelong Agent for Strategy Self-Exploration to Jailbreak LLMscited by 199
  4. 2024Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Makingcited by 144
  5. 2023The Troubling Emergence of Hallucination in Large Language Models -- An Extensive Definition, Quantification, and Prescriptive Remediationscited by 131
  6. 2023HalluciDoctor: Mitigating Hallucinatory Toxicity in Visual Instruction Datacited by 126
  7. 2023Language Models Meet World Models: Embodied Experiences Enhance Language Modelscited by 124
  8. 2023Do Language Models Know When They're Hallucinating References?cited by 116
  9. 2023Can an Embodied Agent Find Your "Cat-shaped Mug"? LLM-Guided Exploration for Zero-Shot Object Navigationcited by 74
  10. 2023"No, to the Right" -- Online Language Corrections for Robotic Manipulation via Shared Autonomycited by 60
  11. 2024A Theoretical Understanding of Self-Correction through In-context Alignmentcited by 44
  12. 2024BadRobot: Jailbreaking Embodied LLM Agents in the Physical Worldcited by 43
  13. 20243D-GRAND: A Million-Scale Dataset for 3D-LLMs with Better Grounding and Less Hallucinationcited by 36
  14. 2024LaMMA-P: Generalizable Multi-Agent Long-Horizon Task Allocation and Planning with LM-Driven PDDL Plannercited by 32
  15. 2024Can We Trust Embodied Agents? Exploring Backdoor Attacks against Embodied LLM-based Decision-Making Systemscited by 27
  16. 2025MMDT: Decoding the Trustworthiness and Safety of Multimodal Foundation Modelscited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 5.

6 papers.

  1. 2026Safety in Embodied AI: A Survey of Risks, Attacks, and Defensescited by 7
  2. 2025AI Must not be Fully Autonomouscited by 3
  3. 20263D-VCD: Hallucination Mitigation in 3D-LLM Embodied Agents through Visual Contrastive Decodingcited by 3
  4. 2026VLN-NF: Feasibility-Aware Vision-and-Language Navigation with False-Premise Instructionscited by 3
  5. 2026MemEvoBench: Benchmarking Safety Risks from Memory Misevolution in LLM Agentscited by 1
  6. 2026STAGE: Diagnosing Semantic Transfer at Grounded Execution in Embodied Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025LLM-based Agents Suffer from Hallucinations: A Survey of Taxonomy, Methods, and Directions
  2. 2025MIRAGE-Bench: LLM Agent is Hallucinating and Where to Find Them
  3. 2025From "Hallucination" to "Suture": Insights from Language Philosophy to Enhance Large Language Models
  4. 2025Mitigating Hallucination in Large Language Models (LLMs): An Application-Oriented Survey on RAG, Reasoning, and Agentic Systems
  5. 2026AgentHallu: Benchmarking Automated Hallucination Attribution of LLM-based Agents
  6. 2026HalluWorld: A Controlled Benchmark for Hallucination via Reference World Models
  7. 2026PRISM: Probing Reasoning, Instruction, and Source Memory in LLM Hallucinations
  8. 2024A Survey on Hallucination in Large Vision-Language Models