mlsanityData to 14 Oct 2026

Hallucination of Multimodal Large Language Models: A Survey

arXiv 2404.18930 ↗First posted 29 Apr 2024cs.CVcited by 388

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

This survey presents a comprehensive analysis of the phenomenon of hallucination in multimodal large language models (MLLMs), also known as Large Vision-Language Models (LVLMs), which have demonstrated significant advancements and remarkable abilities in multimodal tasks. Despite these promising developments, MLLMs often generate outputs that are inconsistent with the visual content, a challenge known as hallucination, which poses substantial obstacles to their practical deployment and raises concerns regarding their reliability in real-world applications. This problem has attracted increasing attention, prompting efforts to detect and mitigate such inaccuracies. We review recent advances in identifying, evaluating, and mitigating these hallucinations, offering a detailed overview of the underlying causes, evaluation benchmarks, metrics, and strategies developed to address this issue. Additionally, we analyze the current challenges and limitations, formulating open questions that delineate potential pathways for future research. By drawing the granular classification and landscapes of hallucination causes, evaluation benchmarks, and mitigation methods, this survey aims to deepen the understanding of hallucinations in MLLMs and inspire further advancements in the field. Through our thorough and in-depth review, we contribute to the ongoing dialogue on enhancing the robustness and reliability of MLLMs, providing valuable insights and resources for researchers and practitioners alike. Resources are available at: https://github.com/showlab/Awesome-MLLM-Hallucination.

Key facts. Cited by 388 papers in the map (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 92 of its 193 references that are in the map. First posted on arXiv 29 Apr 2024.

Builds on

92 of 193 references.

Closest ancestors · 19

Same direction, newest first

  1. 2024Detecting and Mitigating Hallucination in Large Vision Language Models via Fine-Grained AI Feedback22 Apr
  2. 2024Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding27 Mar
  3. 2024Multi-Modal Hallucination Control by Visual Information Grounding20 Mar
  4. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding1 Mar
  5. 2024IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decoding28 Feb
  6. 2024EFUF: Efficient Fine-grained Unlearning Framework for Mitigating Hallucinations in Multimodal Large Language Models15 Feb
  7. 2023Mitigating Fine-Grained Hallucination by Fine-Tuning Large Vision-Language Models with Caption Rewrites4 Dec
  8. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback1 Dec
  9. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  10. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  11. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization28 Nov
  12. 2023Mitigating Hallucination in Visual Language Models with Visual Supervision27 Nov
7 more
  1. 2023HalluciDoctor: Mitigating Hallucinatory Toxicity in Visual Instruction Data22 Nov
  2. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation13 Nov
  3. 2023Holistic Analysis of Hallucination in GPT-4V(ision): Bias and Interference Challenges6 Nov
  4. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Models1 Oct
  5. 2023CIEM: Contrastive Instruction Evaluation Method for Better Instruction Tuning5 Sep
  6. 2023Evaluation and Analysis of Hallucination in Large Vision-Language Models29 Aug
  7. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun

Other references · 73

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2017Proximal Policy Optimization Algorithmscited by 15,215
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  7. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  8. 2023Segment Anythingcited by 8,234
  9. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  10. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  11. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  12. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
38 more
  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  2. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  3. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  4. 2017Deep reinforcement learning from human preferencescited by 3,912
  5. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  6. 2015Fast R-CNNcited by 2,933
  7. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  8. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  9. 2020Learning to summarize from human feedbackcited by 2,588
  10. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  11. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  12. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  13. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  14. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  15. 2023LIMA: Less Is More for Alignmentcited by 1,100
  16. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  17. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  18. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
  19. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  20. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  21. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 735
  22. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  23. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 708
  24. 2023Instruction Tuning with GPT-4cited by 686
  25. 2018Object Hallucination in Image Captioningcited by 669
  26. 2023Instruction Tuning for Large Language Models: A Surveycited by 623
  27. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 610
  28. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 609
  29. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  30. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 550
  31. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 547
  32. 2023Benchmarking Large Language Models for News Summarizationcited by 445
  33. 2022Knowledge Unlearning for Mitigating Privacy Risks in Language Modelscited by 437
  34. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 376
  35. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 334
  36. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 302
  37. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 286
  38. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255

50 most cited of 73 All 73 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 81; 2025: 177; 2026: 130.

388 papers.

Sort:

By area:

  1. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
  2. 2025MMaDA: Multimodal Large Diffusion Language Modelscited by 250
  3. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
  4. 2024RLAIF-V: Open-Source AI Feedback Leads to Super GPT-4V Trustworthinesscited by 131
  5. 2024One Token to Seg Them All: Language Instructed Reasoning Segmentation in Videoscited by 100
  6. 2024MLLM can see? Dynamic Correction Decoding for Hallucination Mitigationcited by 88
  7. 2024SPA-VL: A Comprehensive Safety Preference Alignment Dataset for Vision Language Modelcited by 85
  8. 2024CoVLA: Comprehensive Vision-Language-Action Dataset for Autonomous Drivingcited by 77
  9. 2024VL-Uncertainty: Detecting Hallucination in Large Vision-Language Model via Uncertainty Estimationcited by 73
  10. 2024Look Twice Before You Answer: Memory-Space Visual Retracing for Hallucination Mitigation in Multimodal Large Language Modelscited by 72
  11. 2024What's in the Image? A Deep-Dive into the Vision of Vision Language Modelscited by 68
  12. 2024V-DPO: Mitigating Hallucination in Large Vision Language Models via Vision-Guided Direct Preference Optimizationcited by 67
38 more
  1. 2025The Hidden Life of Tokens: Reducing Hallucination of Large Vision-Language Models via Visual Information Steeringcited by 66
  2. 2024Benchmarking Multimodal Retrieval Augmented Generation with Dynamic VQA Dataset and Self-adaptive Planning Agentcited by 65
  3. 2025Mitigating Hallucinations in Large Vision-Language Models via DPO: On-Policy Data Hold the Keycited by 65
  4. 2026MIRAGE: The Illusion of Visual Understandingcited by 65
  5. 2025Scaling Text-Rich Image Understanding via Code-Guided Synthetic Multimodal Data Generationcited by 58
  6. 2024Detecting and Evaluating Medical Hallucinations in Large Vision Language Modelscited by 57
  7. 2024Visual Prompting in Multimodal Large Language Models: A Surveycited by 54
  8. 2024MultiTrust: A Comprehensive Benchmark Towards Trustworthy Multimodal Large Language Modelscited by 53
  9. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  10. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challengescited by 47
  11. 2024Nullu: Mitigating Object Hallucinations in Large Vision-Language Models via HalluSpace Projectioncited by 46
  12. 2024Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Modelscited by 44
  13. 2025Self-Correcting Decoding with Generative Feedback for Mitigating Hallucinations in Large Vision-Language Modelscited by 44
  14. 2024Mitigating Modality Prior-Induced Hallucinations in Multimodal Large Language Models via Deciphering Attention Causalitycited by 42
  15. 2024CODE: Contrasting Self-generated Description to Combat Hallucination in Large Multi-modal Modelscited by 38
  16. 2024Core Knowledge Deficits in Multi-Modal Language Modelscited by 38
  17. 2025Reasoning about Uncertainty: Do Reasoning Models Know When They Don't Know?cited by 38
  18. 2024Generalized Out-of-Distribution Detection and Beyond in Vision Language Model Era: A Surveycited by 36
  19. 2024MovieBench: A Hierarchical Movie Level Dataset for Long Video Generationcited by 36
  20. 2024X2-DFD: A framework for eXplainable and eXtendable Deepfake Detectioncited by 35
  21. 2024Measuring and Enhancing Trustworthiness of LLMs in RAG through Grounded Attributions and Learning to Refusecited by 34
  22. 2024Visual Description Grounding Reduces Hallucinations and Boosts Reasoning in LVLMscited by 33
  23. 2025Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoningcited by 33
  24. 2025Generate, but Verify: Reducing Hallucination in Vision-Language Models with Retrospective Resamplingcited by 33
  25. 2025VideoHallu: Evaluating and Mitigating Multi-modal Hallucinations on Synthetic Video Understandingcited by 32
  26. 2025CHiP: Cross-modal Hierarchical Direct Preference Optimization for Multimodal LLMscited by 31
  27. 2024Unveiling the Ignorance of MLLMs: Seeing Clearly, Answering Incorrectlycited by 30
  28. 2024ICT: Image-Object Cross-Level Trusted Intervention for Mitigating Object Hallucination in Large Vision-Language Modelscited by 30
  29. 2025Can Large Vision Language Models Read Maps Like a Human?cited by 30
  30. 2024VideoQA in the Era of LLMs: An Empirical Studycited by 29
  31. 2024Thinking Before Looking: Improving Multimodal LLM Reasoning via Mitigating Visual Hallucinationcited by 29
  32. 2025Re-Align: Aligning Vision Language Models via Retrieval-Augmented Direct Preference Optimizationcited by 29
  33. 2024EMO-LLaMA: Enhancing Facial Emotion Understanding with Instruction Tuningcited by 28
  34. 2025Recognition through Reasoning: Reinforcing Image Geo-localization with Large Vision-Language Modelscited by 28
  35. 2025Mitigating Object Hallucinations via Sentence-Level Early Interventioncited by 28
  36. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  37. 2025A Survey of Multimodal Hallucination Evaluation and Detectioncited by 27
  38. 2024Securing Large Language Models: Addressing Bias, Misinformation, and Prompt Attackscited by 26

50 most cited of 388 All 388 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Hallucination in Multimodal Foundation Models: A Survey on Causes, Corrections, and Evaluations
  2. 2024A Survey on Hallucination in Large Vision-Language Models
  3. 2023Evaluating Object Hallucination in Large Vision-Language Models
  4. 2024Mitigating Multilingual Hallucination in Large Vision-Language Models
  5. 2024Unified Hallucination Detection for Multimodal Large Language Models
  6. 2024Hallu-PI: Evaluating Hallucination in Multi-modal Large Language Models within Perturbed Inputs
  7. 2024Piculet: Specialized Models-Guided Hallucination Decrease for MultiModal Large Language Models
  8. 2024Skip n: A Simple Method to Reduce Hallucination in Large Vision-Language Models