mlsanityData to 7 Oct 2026

FADE: Mitigating Hallucinations by Reducing Language-Prior Dominance in Large Vision-Language Models

arXiv 2606.29431 ↗First posted 28 Jun 2026cs.AIcited by 2

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Despite the impressive capabilities of Large Vision-Language Models (LVLMs), they remain susceptible to hallucination, generating content inconsistent with the input image. Recent studies attribute this to the dominance of language priors over visual inputs and employ contrastive decoding methods to mitigate this dominance, but the mechanistic origin remains unexplored. We investigate the information flow through each transformer layer and find that attention modules consistently aggregate visual evidence, while FFN modules at critical layers act as the source of language priors. These priors can override visual evidence, causing correct predictions in intermediate layers to drift toward incorrect outputs. Based on this insight, we propose FADE (FFN Attenuation for DEcoding), a training-free method that attenuates FFN outputs to reduce language-prior dominance. Evaluations on POPE, CHAIR, and MME benchmarks across LLaVA-1.5, mPLUG-Owl2, and InstructBLIP show that FADE effectively mitigates hallucinations while preserving inference efficiency.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 35 of its 49 references that are in the map. First posted on arXiv 28 Jun 2026.

Builds on

35 of 49 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025Intervene-All-Paths: Unified Mitigation of LVLM Hallucinations across Alignment Formats21 Nov
  2. 2025MAP: Mitigating Hallucinations in Large Vision-Language Models with Map-Level Attention Processing3 Aug
  3. 2025The Hidden Life of Tokens: Reducing Hallucination of Large Vision-Language Models via Visual Information Steering5 Feb
  4. 2024Mitigating Hallucinations in Large Vision-Language Models (LVLMs) via Language-Contrastive Decoding (LCD)6 Aug
  5. 2024Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Models4 Aug
  6. 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMs31 Jul
  7. 2024Mitigating Object Hallucinations in Large Vision-Language Models with Assembly of Global and Local Attention18 Jun
  8. 2024CODE: Contrasting Self-generated Description to Combat Hallucination in Large Multi-modal Models4 Jun
  9. 2024Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding27 Mar
  10. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  11. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  12. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization28 Nov
1 more
  1. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun

Other references · 22

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  4. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  6. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  7. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  8. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  9. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  10. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  11. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  12. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
10 more
  1. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  2. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  3. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  4. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 522
  5. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 288
  6. 2024ReDeEP: Detecting Hallucination in Retrieval-Augmented Generation via Mechanistic Interpretabilitycited by 100
  7. 2026Chain-of-Thought Compression Should Not Be Blind: V-Skip for Efficient Multimodal Reasoning via Dual-Path Anchoringcited by 21
  8. 2026PointCoT: A Multi-modal Benchmark for Explicit 3D Geometric Reasoningcited by 19
  9. 2026Not All Queries Need Deep Thought: CoFiCot for Adaptive Coarse-to-fine Stateful Refinementcited by 13
  10. 2025Not All Errors Are Created Equal: ASCoT Addresses Late-Stage Fragility in Efficient LLM Reasoningcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026SPARK: Susceptibility-Guided Profiling and Steering of Latent Reasoning States in Large Language Modelscited by 0
  2. 2026Better Starts, Better Ends: Bootstrapped Iterative Self-Reasoning Distillation for Compressed Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Mitigating Hallucinations in Large Vision-Language Models via Summary-Guided Decoding
  2. 2024VaLiD: Mitigating the Hallucination of Large Vision Language Models by Visual Layer Fusion Contrastive Decoding
  3. 2025MINT: Mitigating Hallucinations in Large Vision-Language Models via Token Reduction
  4. 2026Mitigating Hallucinations in Large Vision-Language Models via Causal Route Gating
  5. 2026Decoding by Perturbation: Mitigating MLLM Hallucinations via Dynamic Textual Perturbation
  6. 2026SDCD: Structure-Disrupted Contrastive Decoding for Mitigating Hallucinations in Large Vision-Language Models
  7. 2025VEGAS: Mitigating Hallucinations in Large Vision-Language Models via Vision-Encoder Attention Guided Adaptive Steering
  8. 2026NoLan: Mitigating Object Hallucinations in Large Vision-Language Models via Dynamic Suppression of Language Priors