mlsanityData to 7 Oct 2026

Mitigating Modality Prior-Induced Hallucinations in Multimodal Large Language Models via Deciphering Attention Causality

arXiv 2410.04780 ↗First posted 7 Oct 2024cs.CVcited by 42

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) have emerged as a central focus in both industry and academia, but often suffer from biases introduced by visual and language priors, which can lead to multimodal hallucination. These biases arise from the visual encoder and the Large Language Model (LLM) backbone, affecting the attention mechanism responsible for aligning multimodal inputs. Existing decoding-based mitigation methods focus on statistical correlations and overlook the causal relationships between attention mechanisms and model output, limiting their effectiveness in addressing these biases. To tackle this issue, we propose a causal inference framework termed CausalMM that applies structural causal modeling to MLLMs, treating modality priors as a confounder between attention mechanisms and output. Specifically, by employing backdoor adjustment and counterfactual reasoning at both the visual and language attention levels, our method mitigates the negative effects of modality priors and enhances the alignment of MLLM's inputs and outputs, with a maximum score improvement of 65.3% on 6 VLind-Bench indicators and 164 points on MME Benchmark compared to conventional methods. Extensive experiments validate the effectiveness of our approach while being a plug-and-play solution. Our code is available at: https://github.com/The-Martyr/CausalMM

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 26 of its 56 references that are in the map. First posted on arXiv 7 Oct 2024.

Builds on

26 of 56 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Look Twice Before You Answer: Memory-Space Visual Retracing for Hallucination Mitigation in Multimodal Large Language Models4 Oct
  2. 2024Reefknot: A Comprehensive Benchmark for Relation Hallucination Evaluation, Analysis and Mitigation in Multimodal Large Language Models18 Aug
  3. 2024Hallucination of Multimodal Large Language Models: A Survey29 Apr
  4. 2024A Survey on Hallucination in Large Vision-Language Models1 Feb

Other references · 22

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  2. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  3. 2023A Survey on Multimodal Large Language Modelscited by 898
  4. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  5. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 367
  6. 2023Causal Reasoning and Large Language Models: Opening a New Frontier for Causalitycited by 348
  7. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  8. 2024VITA: Towards Open-Source Interactive Omni Multimodal LLMcited by 172
  9. 2024Efficient Multimodal Large Language Models: A Surveycited by 95
  10. 2023Causal Order: The Key to Leveraging Imperfect Experts in Causal Inferencecited by 46
  11. 2024VLind-Bench: Measuring Language Priors in Large Vision-Language Modelscited by 44
  12. 2024Quantifying and Mitigating Unimodal Biases in Multimodal Large Language Models: A Causal Perspectivecited by 39
10 more
  1. 2023CUTS: Neural Causal Discovery from Irregular Time-Series Datacited by 32
  2. 2022Rhino: Deep Causal Temporal Relationship Learning With History-dependent Noisecited by 30
  3. 2020Causality Learning: A New Perspective for Interpretable Machine Learningcited by 28
  4. 2024Large Language Models and Causal Inference in Collaboration: A Surveycited by 26
  5. 2024MMNeuron: Discovering Neuron-Level Domain-Specific Interpretation in Multimodal Large Language Modelcited by 26
  6. 2023Causal reasoning in typical computer vision taskscited by 12
  7. 2022Predicting Cellular Responses with Variational Causal Inference and Refined Relational Informationcited by 11
  8. 2024GeoReasoner: Reasoning On Geospatially Grounded Context For Natural Language Understandingcited by 10
  9. 2023An Empirical Study on the Language Modal in Visual Question Answeringcited by 3
  10. 2024Causal-StoNet: Causal Inference for High-Dimensional Complex Datacited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 18; 2026: 21.

42 papers.

Sort:

By area:

  1. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  2. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  3. 2025Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoningcited by 33
  4. 2025Evaluating and Steering Modality Preferences in Multimodal Large Language Modelcited by 33
  5. 2025SafeEraser: Enhancing Safety in Multimodal Large Language Models through Multimodal Machine Unlearningcited by 31
  6. 2025MathAgent: Leveraging a Mixture-of-Math-Agent Framework for Real-World Multimodal Mathematical Error Detectioncited by 24
  7. 2024VidHal: Benchmarking Temporal Hallucinations in Vision LLMscited by 19
  8. 2025DocPruner: A Storage-Efficient Framework for Multi-Vector Visual Document Retrieval via Adaptive Patch-Level Embedding Pruningcited by 19
  9. 2025Diagnosing and Mitigating Modality Interference in Multimodal Large Language Modelscited by 18
  10. 2025Mitigating Visual Knowledge Forgetting in MLLM Instruction-tuning via Modality-decoupled Gradient Descentcited by 13
  11. 2026Hallucination Begins Where Saliency Dropscited by 8
  12. 2025EssayJudge: A Multi-Granular Benchmark for Assessing Automated Essay Scoring Capabilities of Multimodal Large Language Modelscited by 7
30 more
  1. 2025CAFES: A Collaborative Multi-Agent Framework for Multi-Granular Multimodal Essay Scoringcited by 7
  2. 2025Intervene-All-Paths: Unified Mitigation of LVLM Hallucinations across Alignment Formatscited by 6
  3. 2025Enhancing Visual Reliance in Text Generation: A Bayesian Perspective on Mitigating Hallucination in Large Vision-Language Modelscited by 5
  4. 2025IKOD: Mitigating Visual Attention Degradation in Large Vision-Language Modelscited by 5
  5. 2026AMPS: Adaptive Modality Preference Steering via Functional Entropycited by 4
  6. 2025Pierce the Mists, Greet the Sky: Decipher Knowledge Overshadowing via Knowledge Circuit Analysiscited by 3
  7. 2025CAI: Caption-Sensitive Attention Intervention for Mitigating Object Hallucination in Large Vision-Language Modelscited by 3
  8. 2025Can Large Multimodal Models Actively Recognize Faulty Inputs? A Systematic Evaluation Framework of Their Input Scrutiny Abilitycited by 3
  9. 2026Dismantling Pathological Shortcuts: A Causal Framework for Faithful LVLM Decodingcited by 3
  10. 2026One Token, Two Fates: A Unified Framework via Vision Token Manipulation Against MLLMs Hallucinationcited by 2
  11. 2026Towards Robust Endogenous Reasoning: Unifying Drift Adaptation in Non-Stationary Tuningcited by 2
  12. 2026Beyond Perceptual Shortcuts: Causal-Inspired Debiasing Optimization for Generalizable Video Reasoning in Lightweight MLLMscited by 2
  13. 2026Mitigating Hallucinations in Large Vision-Language Models via Causal Route Gatingcited by 2
  14. 2025Noise Augmented Fine Tuning for Mitigating Hallucinations in Large Language Modelscited by 1
  15. 2026Generalizable Multimodal Large Language Model Editing via Invariant Trajectory Learningcited by 1
  16. 2026EnsemHalDet: Robust VLM Hallucination Detection via Ensemble of Internal State Detectorscited by 1
  17. 2026ART: Attention Replacement Technique to Improve Factuality in LLMscited by 1
  18. 2026Rethinking Entropy Allocation in LLM-based ASR: Understanding the Dynamics between Speech Encoders and LLMscited by 1
  19. 2026NIM4-ASR: Towards Efficient, Robust, and Customizable Real-Time LLM-Based ASRcited by 1
  20. 2026RAVE: Re-Allocating Visual Attention in Large Multimodal Modelscited by 1
  21. 2024OccludeNet: A Causal Journey into Mixed-View Actor-Centric Video Action Recognition under Occlusionscited by 0
  22. 2025Intervening in Black Box: Concept Bottleneck Model for Enhancing Human Neural Network Mutual Understandingcited by 0
  23. 2026System-Mediated Attention Imbalances Make Vision-Language Models Say Yescited by 0
  24. 2026ClueTracer: Question-to-Vision Clue Tracing for Training-Free Hallucination Suppression in Multimodal Reasoningcited by 0
  25. 2026CAST: Mitigating Object Hallucination in Large Vision-Language Models via Caption-Guided Visual Attention Steeringcited by 0
  26. 2026ChronoPhyBench: Do MLLMs Truly Understand the World or Merely Exploit Language Priors?cited by 0
  27. 2026When Model Priors Conflict with Visual Evidence: Mitigating Commonsense-Driven Hallucinations by Selective Prior Calibrationcited by 0
  28. 2026Do Multimodal LLMs See Before They Read? Diagnosing Contextual Sycophancycited by 0
  29. 2026MLLMs Hallucinate when Information Distribution Drifts in Synergy Headscited by 0
  30. 2026Devils in Question Relay: Source-Conditioned Relay Steering to Mitigate Hallucinations in Audio-visual Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Causal-LLaVA: Causal Disentanglement for Mitigating Hallucination in Multimodal Large Language Models
  2. 2025Modality Bias in LVLMs: Analyzing and Mitigating Object Hallucination via Attention Lens
  3. 2025Treble Counterfactual VLMs: A Causal Approach to Hallucination
  4. 2024The Curse of Multi-Modalities: Evaluating Hallucinations of Large Multimodal Models across Language, Visual, and Audio
  5. 2026Causal Decoding for Hallucination-Resistant Multimodal Large Language Models
  6. 2026Decoding by Perturbation: Mitigating MLLM Hallucinations via Dynamic Textual Perturbation
  7. 2025Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs
  8. 2025Robust Multimodal Large Language Models Against Modality Conflict