mlsanityData to 13 Oct 2026

Hallucination in Multimodal Foundation Models: A Survey on Causes, Corrections, and Evaluations

arXiv 2410.15359 ↗First posted 20 Oct 2024cs.AIcited by 8

Not on the map. Why?

Abstract (arXiv)

Multimodal Foundation Models represent a significant leap in artificial intelligence. Among them, Large Vision-Language Models (LVLMs) serve as the typical representative of these foundation models, which integrate visual modality directly into Large Language Models (LLMs). They have demonstrated strong capabilities in information processing and generation. However, the existence of hallucinations has limited the potential and practical effectiveness of LVLM in various fields. Although lots of work has been devoted to hallucination mitigation and correction, there are few reviews to summarize them. To address this gap, this survey provides a systematic review of the hallucination landscape in LVLMs. We categorize the causes related to model architecture and data quality, and construct a comprehensive taxonomy of existing mitigation strategies. Furthermore, we critically assess current hallucination evaluation benchmarks from both discriminative and generative perspectives, highlighting the limitations of existing metrics. This survey concludes by discussing open challenges and future research directions to advance the reliability and trustworthiness of LVLMs.

Key facts. Cited by 8 papers in the map (data through 13 Oct 2026). Builds on 49 of its 85 references that are in the map. First posted on arXiv 20 Oct 2024.

Builds on

49 of 85 references.

References · 49

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,043
  2. 2023GPT-4 Technical Reportcited by 18,278
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,550
  4. 2022Training language models to follow instructions with human feedbackcited by 15,856
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,072
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,979
  7. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,492
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,986
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,317
  10. 2023Improved Baselines with Visual Instruction Tuningcited by 4,389
  11. 2021Zero-Shot Text-to-Image Generationcited by 4,284
  12. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,299
37 more
  1. 2022Scaling Instruction-Finetuned Language Modelscited by 3,007
  2. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,864
  3. 2020Learning to summarize from human feedbackcited by 2,575
  4. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,296
  5. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 723
  6. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 708
  7. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 689
  8. 2018Object Hallucination in Image Captioningcited by 663
  9. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  10. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 472
  11. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 442
  12. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 407
  13. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Modelscited by 315
  14. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluationcited by 296
  15. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimizationcited by 223
  16. 2022MAT: Mask-Aware Transformer for Large Hole Image Inpaintingcited by 217
  17. 2023HalluciDoctor: Mitigating Hallucinatory Toxicity in Visual Instruction Datacited by 126
  18. 2023ADriver-I: A General World Model for Autonomous Drivingcited by 103
  19. 2024Mementos: A Comprehensive Benchmark for Multimodal Large Language Model Reasoning over Image Sequencescited by 93
  20. 2024The All-Seeing Project V2: Towards General Relation Comprehension of the Open Worldcited by 90
  21. 2023VIGC: Visual Instruction Generation and Correctioncited by 80
  22. 2023A Survey on Uncertainty Quantification Methods for Deep Learningcited by 79
  23. 2023CIEM: Contrastive Instruction Evaluation Method for Better Instruction Tuningcited by 69
  24. 2023FACTUAL: A Benchmark for Faithful and Consistent Textual Scene Graph Parsingcited by 50
  25. 2024How Easy is It to Fool Your Multimodal LLMs? An Empirical Analysis on Deceptive Promptscited by 47
  26. 2024PhD: A ChatGPT-Prompted Visual hallucination Evaluation Datasetcited by 45
  27. 2021PP-OCRv2: Bag of Tricks for Ultra Lightweight OCR Systemcited by 39
  28. 2024Insight Over Sight: Exploring the Vision-Knowledge Conflicts in Multimodal LLMscited by 37
  29. 2023OphGLM: Training an Ophthalmology Large Language-and-Vision Assistant based on Instructions and Dialoguecited by 34
  30. 2024Cantor: Inspiring Multimodal Chain-of-Thought of MLLMcited by 34
  31. 2023Behind the Magic, MERLIM: Multi-modal Evaluation Benchmark for Large Image-Language Modelscited by 23
  32. 2024Towards Trustworthy AI: A Review of Ethical and Robust Large Language Modelscited by 22
  33. 2023MLLMs-Augmented Visual-Language Representation Learningcited by 21
  34. 2024MM-SAP: A Comprehensive Benchmark for Assessing Self-Awareness of Multimodal Large Language Models in Perceptioncited by 21
  35. 2024EFUF: Efficient Fine-grained Unlearning Framework for Mitigating Hallucinations in Multimodal Large Language Modelscited by 16
  36. 2024Pensieve: Retrospect-then-Compare Mitigates Visual Hallucinationcited by 14
  37. 2024Fact :Teaching MLLMs with Faithful, Concise and Transferable Rationalescited by 7

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2025: 7; 2026: 1.

8 papers.

  1. 2025A Survey of Multimodal Hallucination Evaluation and Detectioncited by 26
  2. 2025The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMscited by 25
  3. 2025VistaDPO: Video Hierarchical Spatial-Temporal Direct Preference Optimization for Large Video Modelscited by 14
  4. 2025Mitigating Object Hallucinations in MLLMs via Multi-Frequency Perturbationscited by 7
  5. 2025Reflective Preference Optimization (RPO): Enhancing On-Policy Alignment via Hint-Guided Reflectioncited by 2
  6. 2025Anatomical Region-Guided Contrastive Decoding: A Plug-and-Play Strategy for Mitigating Hallucinations in Medical VLMscited by 2
  7. 2026VideoHEDGE: Entropy-Based Hallucination Detection for Video-VLMs via Semantic Clustering and Spatiotemporal Perturbationscited by 1
  8. 2025Instruction-Aligned Visual Attention for Mitigating Hallucinations in Large Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A Survey on Hallucination in Large Vision-Language Models
  2. 2024Hallucination of Multimodal Large Language Models: A Survey
  3. 2025A Comprehensive Analysis for Visual Object Hallucination in Large Vision-Language Models
  4. 2026What Makes LVLMs Hallucinate Less? Unveiling the Architectural Factors Behind Hallucination Robustness
  5. 2023Evaluation and Analysis of Hallucination in Large Vision-Language Models
  6. 2025HalluShift++: Bridging Language and Vision through Internal Representation Shifts for Hierarchical Hallucinations in MLLMs
  7. 2023Evaluating Object Hallucination in Large Vision-Language Models
  8. 2026Reducing Object Hallucination in LVLMs via Emphasizing Image-negative Tokens