mlsanityData to 7 Oct 2026

The Curse of Multi-Modalities: Evaluating Hallucinations of Large Multimodal Models across Language, Visual, and Audio

arXiv 2410.12787 ↗First posted 16 Oct 2024cs.CVcited by 48

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Recent advancements in large multimodal models (LMMs) have significantly enhanced performance across diverse tasks, with ongoing efforts to further integrate additional modalities such as video and audio. However, most existing LMMs remain vulnerable to hallucinations, the discrepancy between the factual multimodal input and the generated textual output, which has limited their applicability in various real-world scenarios. This paper presents the first systematic investigation of hallucinations in LMMs involving the three most common modalities: language, visual, and audio. Our study reveals two key contributors to hallucinations: overreliance on unimodal priors and spurious inter-modality correlations. To address these challenges, we introduce the benchmark The Curse of Multi-Modalities (CMM), which comprehensively evaluates hallucinations in LMMs, providing a detailed analysis of their underlying issues. Our findings highlight key vulnerabilities, including imbalances in modality integration and biases from training data, underscoring the need for balanced cross-modal learning and enhanced hallucination mitigation strategies. Based on our observations and findings, we suggest potential research directions that could enhance the reliability of LMMs.

Key facts. Cited by 48 papers in the map (data through 7 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 55 of its 68 references that are in the map. First posted on arXiv 16 Oct 2024.

Builds on

55 of 68 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024BEAF: Observing BEfore-AFter Changes to Evaluate Hallucination in Vision-language Models18 Jul
  2. 2024VideoHallucer: Evaluating Intrinsic and Extrinsic Hallucinations in Large Video-Language Models24 Jun
  3. 2024Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding27 Mar
  4. 2024Unified Hallucination Detection for Multimodal Large Language Models5 Feb
  5. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback1 Dec
  6. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  7. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  8. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models23 Oct
  9. 2023Negative Object Presence Evaluation (NOPE) to Measure Object Hallucination in Vision-Language Models9 Oct
  10. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Models1 Oct
  11. 2023Evaluation and Analysis of Hallucination in Large Vision-Language Models29 Aug
  12. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun
1 more
  1. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 42

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  6. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  7. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  8. 2023Mistral 7Bcited by 3,065
  9. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  10. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  11. 2024Qwen2 Technical Reportcited by 2,076
  12. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
30 more
  1. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  2. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  3. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  4. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  5. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  6. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  7. 2018Object Hallucination in Image Captioningcited by 661
  8. 2024Qwen2-Audio Technical Reportcited by 543
  9. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 492
  10. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 488
  11. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  12. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 386
  13. 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 352
  14. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 330
  15. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioningcited by 320
  16. 2024CogVLM2: Visual Language Models for Image and Video Understandingcited by 214
  17. 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modelingcited by 213
  18. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  19. 2023OneLLM: One Framework to Align All Modalities with Languagecited by 185
  20. 2024Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilitiescited by 183
  21. 2024GAMA: A Large Audio-Language Model with Advanced Audio Understanding and Complex Reasoning Abilitiescited by 151
  22. 2022Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questionscited by 116
  23. 2024GroundingGPT:Language Enhanced Multi-modal Grounding Modelcited by 101
  24. 2024video-SALMONN: Speech-Enhanced Audio-Visual Large Language Modelscited by 101
  25. 2024The All-Seeing Project V2: Towards General Relation Comprehension of the Open Worldcited by 90
  26. 2021Let there be a clock on the beach: Reducing Object Hallucination in Image Captioningcited by 74
  27. 2024Reka Core, Flash, and Edge: A Series of Powerful Multimodal Language Modelscited by 67
  28. 2023Auto-ACD: A Large-scale Dataset for Audio-Language Representation Learningcited by 32
  29. 2023Fine-grained Audio-Visual Joint Representations for Multimodal Large Language Modelscited by 21
  30. 2024On the Audio Hallucinations in Large Audio-Video Language Modelscited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 20; 2026: 26.

48 papers.

Sort:

By area:

  1. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  2. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
  3. 2025Towards Holistic Evaluation of Large Audio-Language Models: A Comprehensive Surveycited by 38
  4. 2025MLLMs are Deeply Affected by Modality Biascited by 38
  5. 2025Evaluating and Steering Modality Preferences in Multimodal Large Language Modelcited by 33
  6. 2025AVCD: Mitigating Hallucinations in Audio-Visual Large Language Models through Contrastive Decodingcited by 26
  7. 2024From Specific-MLLMs to Omni-MLLMs: A Survey on MLLMs Aligned with Multi-modalitiescited by 22
  8. 2025Cross-Modal Attention Calibration for LVLM Hallucination Mitigationcited by 21
  9. 2025AudioLens: A Closer Look at Auditory Attribute Perception of Large Audio-Language Modelscited by 18
  10. 2026Audio Flamingo Next: Next-Generation Open Audio-Language Models for Speech, Sound, and Musiccited by 15
  11. 2025VistaDPO: Video Hierarchical Spatial-Temporal Direct Preference Optimization for Large Video Modelscited by 14
  12. 2025MCA-LLaVA: Manhattan Causal Attention for Reducing Hallucination in Large Vision-Language Modelscited by 11
36 more
  1. 2025MindGPT-4ov: An Enhanced MLLM via a Multi-Stage Post-Training Paradigmcited by 9
  2. 2025OmniDPO: A Preference Optimization Framework to Address Omni-Modal Hallucinationcited by 8
  3. 2025CCHall: A Novel Benchmark for Joint Cross-Lingual and Cross-Modal Hallucinations Detection in Large Language Modelscited by 7
  4. 2025Fork-Merge Decoding: Enhancing Multimodal Understanding in Audio-Visual Large Language Modelscited by 7
  5. 2025SpurLens: Automatic Detection of Spurious Cues in Multimodal LLMscited by 6
  6. 2026Instruction Anchor: Dissecting the Mechanistic Dynamics of Modality Arbitrationcited by 6
  7. 2026Beyond Unimodal Shortcuts: MLLMs as Cross-Modal Reasoners for Grounded Named Entity Recognitioncited by 6
  8. 2024Seeing is Deceiving: Exploitation of Visual Pathways in Multi-Modal Language Modelscited by 4
  9. 2025GHOST: Hallucination-Inducing Image Generation for Multimodal LLMscited by 4
  10. 2026Don't Let the Video Speak: Audio-Contrastive Preference Optimization for Audio-Visual Language Modelscited by 4
  11. 2026TangramPuzzle: Evaluating Multimodal Large Language Models with Compositional Spatial Reasoningcited by 3
  12. 2026MoD-DPO: Towards Mitigating Cross-modal Hallucinations in Omni LLMs using Modality Decoupled Preference Optimizationcited by 3
  13. 2026Dismantling Pathological Shortcuts: A Causal Framework for Faithful LVLM Decodingcited by 3
  14. 2025Exploring Implicit Visual Misunderstandings in Multimodal Large Language Models through Attention Analysiscited by 2
  15. 2025Unleashing the Intrinsic Visual Representation Capability of Multimodal Large Language Modelscited by 2
  16. 2026MAD: Modality-Adaptive Decoding for Mitigating Cross-Modal Hallucinations in Multimodal Large Language Modelscited by 2
  17. 2026Senses Wide Shut: A Representation-Action Gap in Omnimodal LLMscited by 2
  18. 2025v-HUB: A Benchmark for Video Humor Understanding from Vision and Soundcited by 1
  19. 2026Distorted or Fabricated? A Survey on Hallucination in Video LLMscited by 1
  20. 2026Separate First, Fuse Later: Mitigating Cross-Modal Interference in Audio-Visual LLMs Reasoning with Modality-Specific Chain-of-Thoughtcited by 1
  21. 2026SVHalluc: Benchmarking Speech-Vision Hallucination in Audio-Visual Large Language Modelscited by 1
  22. 2026C3PO: Evaluating Cross-Modal Composition and Counterfactual Performance in Omnimodal Modelscited by 1
  23. 2026Multi-turn Conversational AI from Text to Multimodal Interaction: Data, Models, Evaluation, and Open Challengescited by 1
  24. 2025Evaluating Hallucinations in Audio-Visual Multimodal LLMs with Spoken Queries under Diverse Acoustic Conditionscited by 0
  25. 2026IRIS: Implicit Reward-Guided Internal Sifting for Mitigating Multimodal Hallucinationcited by 0
  26. 2026Localizing and Editing Knowledge in Large Audio-Language Modelscited by 0
  27. 2026Chain of Modality: From Static Fusion to Dynamic Orchestration in Omni-MLLMscited by 0
  28. 2026YARD: Y-Architecture Register Decoding for Efficient Hallucination Mitigation in Large Vision-Language Modelscited by 0
  29. 2026Speech Meets ELF: Audio Conditional Continuous-Target Diffusion for Speech Recognition and Translationcited by 0
  30. 2026Noisy-Channel Minimum Bayes Risk Decodingcited by 0
  31. 2026CAER: Conflict-Aware Evidence Routing with Dual Prefix Experts for Multimodal Large Language Modelscited by 0
  32. 2026Evolving Safety Landscape of Multi-modal Large Language Models: A Survey of Emerging Threats and Safeguardscited by 0
  33. 2026Compositional Failure in Audio-Visual LLMs: Late-Layer Prior Dominance Under Cross-modal Conflictcited by 0
  34. 2026Tri-PvP: Exposing Modality Bias in Omni-Modal Large Language Models through Perceptual-Propositional Evidence Conflictscited by 0
  35. 2026Don't Read the Log: Execution Traces Contaminate Verifiers in Video-Generation Agentscited by 0
  36. 2026Devils in Question Relay: Source-Conditioned Relay Steering to Mitigate Hallucinations in Audio-visual Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024AVHBench: A Cross-Modal Hallucination Benchmark for Audio-Visual Large Language Models
  2. 2024Hallucination in Multimodal Foundation Models: A Survey on Causes, Corrections, and Evaluations
  3. 2025Robust Multimodal Large Language Models Against Modality Conflict
  4. 2024Mitigating Modality Prior-Induced Hallucinations in Multimodal Large Language Models via Deciphering Attention Causality
  5. 2024Less is More: Mitigating Multimodal Hallucination from an EOS Decision Perspective
  6. 2025INTER: Mitigating Hallucination in Large Vision-Language Models by Interaction Guidance Sampling
  7. 2025Modality Bias in LVLMs: Analyzing and Mitigating Object Hallucination via Attention Lens
  8. 2024Combating Multimodal LLM Hallucination via Bottom-Up Holistic Reasoning