mlsanityData to 14 Oct 2026

A Comprehensive Survey of Hallucination in Large Language, Image, Video and Audio Foundation Models

arXiv 2405.09589 ↗First posted 15 May 2024cs.LG cs.AI cs.CLcited by 100

Not on the map. Why?

Cited by 20 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The rapid advancement of foundation models (FMs) across language, image, audio, and video domains has shown remarkable capabilities in diverse tasks. However, the proliferation of FMs brings forth a critical challenge: the potential to generate hallucinated outputs, particularly in high-stakes applications. The tendency of foundation models to produce hallucinated content arguably represents the biggest hindrance to their widespread adoption in real-world scenarios, especially in domains where reliability and accuracy are paramount. This survey paper presents a comprehensive overview of recent developments that aim to identify and mitigate the problem of hallucination in FMs, spanning text, image, video, and audio modalities. By synthesizing recent advancements in detecting and mitigating hallucination across various modalities, the paper aims to provide valuable insights for researchers, developers, and practitioners. Essentially, it establishes a clear framework encompassing definition, taxonomy, and detection strategies for addressing hallucination in multimodal foundation models, laying the foundation for future research in this pivotal area.

Key facts. Cited by 100 papers in the map (data through 14 Oct 2026). Builds on 65 of its 100 references that are in the map. First posted on arXiv 15 May 2024.

Builds on

65 of 100 references.

References · 65

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
  3. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  4. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 735
  5. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  6. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 688
  7. 2023The Internal State of an LLM Knows When It's Lyingcited by 659
  8. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 619
  9. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 478
  10. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 445
  11. 2023A Survey of Hallucination in Large Foundation Modelscited by 368
  12. 2023Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedbackcited by 353
38 more
  1. 2023How Language Model Hallucinations Can Snowballcited by 311
  2. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  3. 2022RARR: Researching and Revising What Language Models Say, Using Language Modelscited by 287
  4. 2023FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarioscited by 251
  5. 2023NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizerscited by 235
  6. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimizationcited by 225
  7. 2023A Stitch in Time Saves Nine: Detecting and Mitigating Hallucinations of LLMs by Validating Low-Confidence Generationcited by 206
  8. 2024Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decodingcited by 203
  9. 2023Sources of Hallucination by Large Language Models on Inference Taskscited by 169
  10. 2023Text-to-Audio Generation using Instruction-Tuned LLM and Latent Diffusion Modelcited by 145
  11. 2022Diffusion Models for Video Prediction and Infillingcited by 135
  12. 2023Hallucinations in Large Multilingual Translation Modelscited by 129
  13. 2023Self-contradictory Hallucinations of Large Language Models: Evaluation, Detection and Mitigationcited by 124
  14. 2023ChartBench: A Benchmark for Complex Visual Reasoning in Chartscited by 120
  15. 2020Multi-modal Dense Video Captioningcited by 113
  16. 2023Generating Benchmarks for Factuality Evaluation of Language Modelscited by 105
  17. 2023MusicLDM: Enhancing Novelty in Text-to-Music Generation Using Beat-Synchronous Mixup Strategiescited by 94
  18. 2023LP-MusicCaps: LLM-Based Pseudo Music Captioningcited by 83
  19. 2023VIGC: Visual Instruction Generation and Correctioncited by 80
  20. 2022Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-trainingcited by 72
  21. 2023CIEM: Contrastive Instruction Evaluation Method for Better Instruction Tuningcited by 69
  22. 2023Natural Language Supervision for General-Purpose Audio Representationscited by 61
  23. 2019Streamlined Dense Video Captioningcited by 57
  24. 2023TouchStone: Evaluating Vision-Language Models by Language Modelscited by 55
  25. 2023Putting People in Their Place: Affordance-Aware Human Insertion into Scenescited by 53
  26. 2023PURR: Efficiently Editing Language Model Hallucinations by Denoising Language Model Corruptionscited by 53
  27. 2024PhD: A ChatGPT-Prompted Visual hallucination Evaluation Datasetcited by 45
  28. 2023CompA: Addressing the Gap in Compositional Reasoning in Audio-Language Modelscited by 43
  29. 2023CLIPSyntel: CLIP and LLM Synergy for Multimodal Question Summarization in Healthcarecited by 43
  30. 2023A New Benchmark and Reverse Validation Method for Passage-level Hallucination Detectioncited by 42
  31. 2023Sparks of Large Audio Models: A Survey and Outlookcited by 41
  32. 2023Deficiency of Large Language Models in Finance: An Empirical Examination of Hallucinationcited by 41
  33. 2023Explaining Legal Concepts with Augmented Large Language Models (GPT-4)cited by 40
  34. 2021EMScore: Evaluating Video Captioning via Coarse-Grained and Fine-Grained Embedding Matchingcited by 36
  35. 2024ViGoR: Improving Visual Grounding of Large Vision Language Models with Fine-Grained Reward Modelingcited by 34
  36. 2023StableLLaVA: Enhanced Visual Instruction Tuning with Synthesized Image-Dialogue Datacited by 33
  37. 2023Creating Trustworthy LLMs: Dealing with Hallucinations in Healthcare AIcited by 33
  38. 2024EnCLAP: Combining Neural Audio Codec and Audio-Text Joint Embedding for Automated Audio Captioningcited by 28

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 56; 2026: 38.

100 papers.

Sort:

By area:

  1. 2024VL-Uncertainty: Detecting Hallucination in Large Vision-Language Model via Uncertainty Estimationcited by 73
  2. 2025Are Reasoning Models More Prone to Hallucination?cited by 64
  3. 2025Large Language Models Hallucination: A Comprehensive Surveycited by 49
  4. 2025ConfTuner: Training Large Language Models to Express Their Confidence Verballycited by 39
  5. 2024ICT: Image-Object Cross-Level Trusted Intervention for Mitigating Object Hallucination in Large Vision-Language Modelscited by 30
  6. 2025Large Vision-Language Model Alignment and Misalignment: A Survey Through the Lens of Explainabilitycited by 28
  7. 2025Koel-TTS: Enhancing LLM based Speech Generation with Preference Alignment and Classifier Free Guidancecited by 27
  8. 2025A Comprehensive Survey on Reinforcement Learning-based Agentic Search: Foundations, Roles, Optimizations, Evaluations, and Applicationscited by 22
  9. 2025Hallucination Detection in LLMs with Topological Divergence on Attention Graphscited by 19
  10. 2025Lost in Transcription, Found in Distribution Shift: Demystifying Hallucination in Speech Foundation Modelscited by 18
  11. 2025Beyond Token Probes: Hallucination Detection via Activation Tensors with ACT-ViTcited by 17
  12. 2025Dr.V: A Hierarchical Perception-Temporal-Cognition Framework to Diagnose Video Hallucination by Fine-grained Spatial-Temporal Groundingcited by 13
38 more
  1. 2025RAG-KG-IL: A Multi-Agent Hybrid Framework for Reducing Hallucinations and Enhancing LLM Reasoning through RAG and Incremental Knowledge Graph Learning Integrationcited by 9
  2. 2025LLLMs: A Data-Driven Survey of Evolving Research on Limitations of Large Language Modelscited by 9
  3. 2025Locus: Agentic Predicate Synthesis for Directed Fuzzingcited by 9
  4. 2025OmniDPO: A Preference Optimization Framework to Address Omni-Modal Hallucinationcited by 9
  5. 2025Mitigating Hallucination in VideoLLMs via Temporal-Aware Activation Engineeringcited by 8
  6. 2024Organizing Unstructured Image Collections using Natural Languagecited by 7
  7. 2025HICD: Hallucination-Inducing via Attention Dispersion for Contrastive Decoding to Mitigate Hallucinations in Large Language Modelscited by 7
  8. 2025How Individual Traits and Language Styles Shape Preferences In Open-ended User-LLM Interaction: A Preliminary Studycited by 7
  9. 2025HyPerAlign: Interpretable Personalized LLM Alignment via Hypothesis Generationcited by 7
  10. 2025Attribution, Citation, and Quotation: A Survey of Evidence-based Text Generation with Large Language Modelscited by 7
  11. 2025Mitigating Hallucinations in Multimodal LLMs via Object-aware Preference Optimizationcited by 7
  12. 2026Query-Aware Flow Diffusion for Graph-Based RAG with Retrieval Guaranteescited by 7
  13. 2024SPES: Spectrogram Perturbation for Explainable Speech-to-Text Generationcited by 6
  14. 2025IC-World: In-Context Generation for Shared World Modelingcited by 6
  15. 2026How Contrastive Decoding Enhances Large Audio Language Modelscited by 6
  16. 2025ZINA: Multimodal Fine-grained Hallucination Detection and Editingcited by 5
  17. 2025ToM-SSI: Evaluating Theory of Mind in Situated Social Interactionscited by 5
  18. 2026Trust Me, Import This: Dependency Steering Attacks via Malicious Agent Skillscited by 5
  19. 2025PolyG: Adaptive Graph Traversal for Diverse GraphRAG Questionscited by 4
  20. 2025Endowing Embodied Agents with Spatial Reasoning Capabilities for Vision-and-Language Navigationcited by 4
  21. 2025EcoSafeRAG: Efficient Security through Context Analysis in Retrieval-Augmented Generationcited by 4
  22. 2025mRAG: Elucidating the Design Space of Multi-modal Retrieval-Augmented Generationcited by 4
  23. 2025Do LVLMs Know What They Know? A Systematic Study of Knowledge Boundary Perception in LVLMscited by 4
  24. 2025CCD: Mitigating Hallucinations in Radiology MLLMs via Clinical Contrastive Decodingcited by 4
  25. 2025LLM Bias Detection and Mitigation through the Lens of Desired Distributionscited by 4
  26. 2025Trustworthy Retrosynthesis: Eliminating Hallucinations with a Diverse Ensemble of Reaction Scorerscited by 4
  27. 2025SAFENLIDB: A Privacy-Preserving Safety Alignment Framework for LLM-based Natural Language Database Interfacescited by 4
  28. 2024Enhancing Adverse Drug Event Detection with Multimodal Dataset: Corpus Creation and Model Developmentcited by 3
  29. 2024Aligning Large Language Models and Geometric Deep Models for Protein Representationcited by 3
  30. 2025Towards EnergyGPT: A Large Language Model Specialized for the Energy Sectorcited by 3
  31. 2025Evaluating Uncertainty Quantification Methods in Argumentative Large Language Modelscited by 3
  32. 2025Lie to Me: Knowledge Graphs for Robust Hallucination Self-Detection in LLMscited by 3
  33. 2026ORCE: Order-Aware Alignment of Verbalized Confidence in Large Language Modelscited by 3
  34. 2026DVAO: Dynamic Variance-adaptive Advantage Optimization for Multi-reward Reinforcement Learningcited by 3
  35. 2026AgentGym2: Benchmarking Large Language Model Agents in De-Idealized Real-World Environmentscited by 3
  36. 2025FactReasoner: A Probabilistic Approach to Long-Form Factuality Assessment for Large Language Modelscited by 2
  37. 2025Data-efficient Meta-models for Evaluation of Context-based Questions and Answers in LLMscited by 2
  38. 2025Towards Self-cognitive Exploration: Metacognitive Knowledge Graph Retrieval Augmented Generationcited by 2

50 most cited of 100 All 100 (JSON)