mlsanityData to 14 Oct 2026

MLLMs Hallucinate when Information Distribution Drifts in Synergy Heads

arXiv 2609.09206 ↗First posted 5 Sep 2026cs.CV cs.CL cs.LGcited by 0

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) often struggle with hallucinations, thus hindering their reliable practical applications. Existing attention-based mitigation methods mainly rely on indirect signals (e.g., attention weights) that fail to accurately reflect the actual information shift underlying hallucination generation. In this paper, we propose HEAL, Head-lEvel information disentAnglement and caLibration for identifying and mitigating hallucinations. HEAL first employs causal noise intervention on multi-head outputs to filter out causally redundant heads. Subsequently, it disentangles information distribution within the remaining heads via the counterfactual Difference-in-Differences, categorizing heads into four types. Through analysis, we observe: hallucinations happen when information distribution drifts away from a healthy equilibrium in synergy heads, not strongly correlated with the quantity or strength of modality-specific heads. Motivated by this insight, HEAL injects dynamic information calibration factors into the value vectors of synergy heads and actively regulates visual-language dependencies, steering the output distribution towards factual evidence. Extensive experiments demonstrate that HEAL effectively reduces hallucinations across multiple MLLMs, offering a simple and interpretable pathway to enhance model trustworthiness.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 60 of its 66 references that are in the map. First posted on arXiv 5 Sep 2026.

Builds on

60 of 66 references.

Closest ancestors · 29

Same direction, newest first

  1. 2026Hallucination Begins Where Saliency Drops28 Jan
  2. 2025Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs12 Nov
  3. 2025A Survey of Multimodal Hallucination Evaluation and Detection25 Jul
  4. 2025ONLY: One-Layer Intervention Sufficiently Mitigates Hallucinations in Large Vision-Language Models1 Jul
  5. 2025When Semantics Mislead Vision: Mitigating Large Multimodal Models Hallucinations in Scene Text Spotting and Understanding5 Jun
  6. 2025Seeing Far and Clearly: Mitigating Hallucinations in MLLMs with Attention Causal Decoding22 May
  7. 2025PerturboLLaVA: Reducing Multimodal Hallucinations with Perturbative Visual Training9 Mar
  8. 2025The Hidden Life of Tokens: Reducing Hallucination of Large Vision-Language Models via Visual Information Steering5 Feb
  9. 2024Cracking the Code of Hallucination in LVLMs with Vision-aware Head Divergence18 Dec
  10. 2024Seeing Clearly by Layer Two: Enhancing Attention Heads to Alleviate Hallucination in LVLMs15 Nov
  11. 2024Mitigating Modality Prior-Induced Hallucinations in Multimodal Large Language Models via Deciphering Attention Causality7 Oct
  12. 2024Look Twice Before You Answer: Memory-Space Visual Retracing for Hallucination Mitigation in Multimodal Large Language Models4 Oct
17 more
  1. 2024ConVis: Contrastive Decoding with Hallucination Visualization for Mitigating Hallucinations in Multimodal Large Language Models25 Aug
  2. 2024Reefknot: A Comprehensive Benchmark for Relation Hallucination Evaluation, Analysis and Mitigation in Multimodal Large Language Models18 Aug
  3. 2024Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Models4 Aug
  4. 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMs31 Jul
  5. 2024DOPRA: Decoding Over-accumulation Penalization and Re-allocation in Specific Weighting Layer21 Jul
  6. 2024Reflective Instruction Tuning: Mitigating Hallucinations in Large Vision-Language Models16 Jul
  7. 2024Mitigating Object Hallucinations in Large Vision-Language Models with Assembly of Global and Local Attention18 Jun
  8. 2024Hallucination of Multimodal Large Language Models: A Survey29 Apr
  9. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding1 Mar
  10. 2024IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decoding28 Feb
  11. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  12. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  13. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization28 Nov
  14. 2023HalluciDoctor: Mitigating Hallucinatory Toxicity in Visual Instruction Data22 Nov
  15. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Models1 Oct
  16. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun
  17. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 31

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023Visual Instruction Tuningcited by 7,869
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2025Qwen2.5-VL Technical Reportcited by 5,587
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  7. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  8. 2025Qwen3-VL Technical Reportcited by 2,793
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  10. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  11. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  12. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
19 more
  1. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  2. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  3. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
  4. 2019Root Mean Square Layer Normalizationcited by 1,480
  5. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
  6. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  7. 2018Object Hallucination in Image Captioningcited by 669
  8. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 547
  9. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 376
  10. 2025Why Language Models Hallucinatecited by 220
  11. 2025See What You Are Told: Visual Attention Sink in Large Multimodal Modelscited by 132
  12. 2023Quantifying & Modeling Multimodal Interactions: An Information Decomposition Frameworkcited by 83
  13. 2023VCoder: Versatile Vision Encoders for Multimodal Large Language Modelscited by 42
  14. 2025SparseMM: Head Sparsity Emerges from Visual Concept Responses in MLLMscited by 20
  15. 2016Measuring multivariate redundant information with pointwise common change in surprisalcited by 13
  16. 2025BLINK-Twice: You see, but do you observe? A Reasoning Benchmark on Visual Perceptioncited by 9
  17. 2026Towards Understanding Modality Interaction in Multimodal Language Models via Partial Information Decompositioncited by 5
  18. 2025Partial Information Rate Decompositioncited by 3
  19. 2025Broadcast Channel Cooperative Gain: An Operational Interpretation of Partial Information Decompositioncited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Not All Tokens and Heads Are Equally Important: Dual-Level Attention Intervention for Hallucination Mitigation
  2. 2025Mitigating Hallucination in Large Vision-Language Models through Aligning Attention Distribution to Information Flow
  3. 2025HalluShift++: Bridging Language and Vision through Internal Representation Shifts for Hierarchical Hallucinations in MLLMs
  4. 2026Decoding by Perturbation: Mitigating MLLM Hallucinations via Dynamic Textual Perturbation
  5. 2024Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lens
  6. 2026VIB-Probe: Detecting and Mitigating Hallucinations in Vision-Language Models via Variational Information Bottleneck
  7. 2026Rethinking Visual Neglect: Steering via Context-Preference for MLLM Hallucination Mitigation
  8. 2026Scalpel: Fine-Grained Alignment of Attention Activation Manifolds via Mixture Gaussian Bridges to Mitigate Multimodal Hallucination