mlsanityData to 14 Oct 2026

Extracting Visual Facts from Intermediate Layers for Mitigating Hallucinations in Multimodal Large Language Models

arXiv 2507.15652 ↗First posted 21 Jul 2025cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) have made significant strides by combining visual recognition and language understanding to generate content that is both coherent and contextually accurate. However, MLLMs continue to struggle with object hallucinations, where models produce seemingly plausible but factually incorrect outputs, including objects that do not exist in the image. Recent work has revealed that the prior knowledge in MLLMs significantly suppresses visual information in deep layers, causing hallucinatory outputs. However, how these priors suppress visual information at the intermediate layer stage in MLLMs remains unclear. We observe that visual factual knowledge and the differences between intermediate-layer prior/original probability distributions show similar evolutionary trends in intermediate layers. Motivated by this, we introduce Decoding by Extracting Visual Facts (EVA), a simple, training-free method that dynamically selects intermediate layers with the most significant visual factual information. By contrasting the output distributions of the selected layer derived from the original input and pure-text input, EVA extracts visual factual knowledge and proportionally incorporates it into the final layer to correct the output logits. Importantly, EVA is model-agnostic, seamlessly integrates with various classic decoding strategies, and is applicable across different MLLMs. We validate EVA on widely-used benchmarks, and the results show that it significantly reduces hallucination rates compared to baseline methods, underscoring its effectiveness in mitigating hallucinations.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 40 of its 43 references that are in the map. First posted on arXiv 21 Jul 2025.

Builds on

40 of 43 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025Mirage in the Eyes: Hallucination Attack on Multi-modal Large Language Models with Only Attention Sink25 Jan
  2. 2024Delve into Visual Contrastive Decoding for Hallucination Mitigation of Large Vision-Language Models9 Dec
  3. 2024MLLM can see? Dynamic Correction Decoding for Hallucination Mitigation15 Oct
  4. 2024Look Twice Before You Answer: Memory-Space Visual Retracing for Hallucination Mitigation in Multimodal Large Language Models4 Oct
  5. 2024HELPD: Mitigating Hallucination of LVLMs by Hierarchical Feedback Learning with Vision-enhanced Penalty Decoding30 Sep
  6. 2024Hallucination of Multimodal Large Language Models: A Survey29 Apr
  7. 2024A Survey on Hallucination in Large Vision-Language Models1 Feb
  8. 2023Hallucination Augmented Contrastive Learning for Multimodal Large Language Model12 Dec
  9. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  10. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  11. 2023HalluciDoctor: Mitigating Hallucinatory Toxicity in Visual Instruction Data22 Nov
  12. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun
1 more
  1. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 27

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  3. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  4. 2023Qwen Technical Reportcited by 3,344
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  6. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  7. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
  8. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  9. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 733
  10. 2018Object Hallucination in Image Captioningcited by 669
  11. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 596
  12. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 376
15 more
  1. 2023A Survey of Hallucination in Large Foundation Modelscited by 368
  2. 2022Confident Adaptive Language Modelingcited by 310
  3. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 290
  4. 2024LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinationscited by 244
  5. 2019Depth-Adaptive Transformercited by 236
  6. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Drivingcited by 233
  7. 2024HuatuoGPT-Vision, Towards Injecting Medical Visual Knowledge into Multimodal LLMs at Scalecited by 200
  8. 2023ChatCAD: Interactive Computer-Aided Diagnosis on Medical Image using Large Language Modelscited by 116
  9. 2024ANOLE: An Open, Autoregressive, Native Large Multimodal Models for Interleaved Image-Text Generationcited by 114
  10. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  11. 2024Knowledge Mechanisms in Large Language Models: A Survey and Perspectivecited by 51
  12. 2024In-Context Sharpness as Alerts: An Inner Representation Perspective for Hallucination Mitigationcited by 45
  13. 2024Insights into LLM Long-Context Failures: When Transformers Know but Don't Tellcited by 36
  14. 2023Advancing Medical Imaging with Language Models: A Journey from N-grams to ChatGPTcited by 28
  15. 2024Llama SLayer 8B: Shallow Layers Hold the Key to Knowledge Injectioncited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024VaLiD: Mitigating the Hallucination of Large Vision Language Models by Visual Layer Fusion Contrastive Decoding
  2. 2025Mitigating Hallucinations in Large Vision-Language Models with Internal Fact-based Contrastive Decoding
  3. 2025ONLY: One-Layer Intervention Sufficiently Mitigates Hallucinations in Large Vision-Language Models
  4. 2026FADE: Mitigating Hallucinations by Reducing Language-Prior Dominance in Large Vision-Language Models
  5. 2024Skip n: A Simple Method to Reduce Hallucination in Large Vision-Language Models
  6. 2025HalluShift++: Bridging Language and Vision through Internal Representation Shifts for Hierarchical Hallucinations in MLLMs
  7. 2025Analyzing and Mitigating Object Hallucination: A Training Bias Perspective
  8. 2026Decoding by Perturbation: Mitigating MLLM Hallucinations via Dynamic Textual Perturbation