mlsanityData to 14 Oct 2026

Mitigating Hallucination in Multimodal LLMs with Layer Contrastive Decoding

arXiv 2509.25177 ↗First posted 29 Sep 2025cs.CVcited by 10

On the map: LLM reasoning → Hallucination detection and mitigation

also in 1 more
Abstract (arXiv)

Multimodal Large Language Models (MLLMs) have shown impressive perception and reasoning capabilities, yet they often suffer from hallucinations -- generating outputs that are linguistically coherent but inconsistent with the context of the input image, including inaccuracies in objects, attributes, and relations. To address this challenge, we propose a simple approach called Layer Contrastive Decoding (LayerCD). Our design is motivated by the observation that shallow visual features are much more likely than deep visual features to cause an MLLM to hallucinate as they only capture biased, low-level information that is insufficient for high-level reasoning. Therefore, LayerCD aims to filter out hallucinations by contrasting the output distributions generated from visual features of different levels, specifically those from the shallow and deep layers of the vision encoder, respectively. We conduct extensive experiments on two hallucination benchmarks and show that LayerCD significantly outperforms current state-of-the-art. The code for LayerCD is available at https://github.com/maifoundations/LayerCD .

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in Hallucination detection and mitigation, one of 17 directions in LLM reasoning. Builds on 45 of its 50 references that are in the map. First posted on arXiv 29 Sep 2025.

Builds on

45 of 50 references.

References · 45

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  7. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  8. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  9. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  10. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  11. 2013Visualizing and Understanding Convolutional Networkscited by 2,290
  12. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
33 more
  1. 2024Qwen2 Technical Reportcited by 2,120
  2. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  3. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  4. 2014How transferable are features in deep neural networks?cited by 1,225
  5. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 900
  6. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
  7. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 803
  8. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 719
  9. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
  10. 2023ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoderscited by 712
  11. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  12. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  13. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 688
  14. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 609
  15. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 478
  16. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 445
  17. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 357
  18. 2024A Survey on Hallucination in Large Vision-Language Modelscited by 357
  19. 2021Entity-Based Knowledge Conflicts in Question Answeringcited by 356
  20. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Modelscited by 316
  21. 2023How Language Model Hallucinations Can Snowballcited by 311
  22. 2020ToTTo: A Controlled Table-To-Text Generation Datasetcited by 268
  23. 2023Detecting and Preventing Hallucinations in Large Vision Language Modelscited by 256
  24. 2017Beam Search Strategies for Neural Machine Translationcited by 232
  25. 2017Challenges in Data-to-Document Generationcited by 226
  26. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimizationcited by 225
  27. 2024Multi-Modal Hallucination Control by Visual Information Groundingcited by 208
  28. 2024Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decodingcited by 203
  29. 2022Factuality Enhanced Language Models for Open-Ended Text Generationcited by 202
  30. 2023Hallucination Augmented Contrastive Learning for Multimodal Large Language Modelcited by 121
  31. 2023Exposing Attention Glitches with Flip-Flop Language Modelingcited by 75
  32. 2023VCoder: Versatile Vision Encoders for Multimodal Large Language Modelscited by 42
  33. 2023BatGPT: A Bidirectional Autoregessive Talker from Generative Pre-trained Transformercited by 23

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 10.

10 papers.

  1. 2026Context Tokens are Anchors: Understanding the Repetition Curse in dMLLMs from an Information Flow Perspectivecited by 6
  2. 2026On the Nature of Attention Sink that Shapes Decoding Strategy in Omni-LLMscited by 6
  3. 2026Dismantling Pathological Shortcuts: A Causal Framework for Faithful LVLM Decodingcited by 4
  4. 2026SDCD: Structure-Disrupted Contrastive Decoding for Mitigating Hallucinations in Large Vision-Language Modelscited by 3
  5. 2026TGIF: Text-Guided Layer Fusion Mitigates Hallucination in Multimodal LLMscited by 2
  6. 2026ADAPT: Attention Dynamics Alignment with Preference Tuning for Faithful MLLMscited by 2
  7. 2026Do We Really Need External Tools to Mitigate Hallucinations? SIRA: Shared-Prefix Internal Reconstruction of Attributioncited by 1
  8. 2026YARD: Y-Architecture Register Decoding for Efficient Hallucination Mitigation in Large Vision-Language Modelscited by 1
  9. 2026MLLMs Get It Right, Then Get It Wrong: Tracing and Correcting Late-Layer Textual Biascited by 0
  10. 2026Reliability-Prioritized Fine-Grained Generation in Multimodal Largecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024VaLiD: Mitigating the Hallucination of Large Vision Language Models by Visual Layer Fusion Contrastive Decoding
  2. 2025MaskCD: Mitigating LVLM Hallucinations by Image Head Masked Contrastive Decoding
  3. 2025ONLY: One-Layer Intervention Sufficiently Mitigates Hallucinations in Large Vision-Language Models
  4. 2024ConVis: Contrastive Decoding with Hallucination Visualization for Mitigating Hallucinations in Multimodal Large Language Models
  5. 2024Delve into Visual Contrastive Decoding for Hallucination Mitigation of Large Vision-Language Models
  6. 2024MLLM can see? Dynamic Correction Decoding for Hallucination Mitigation
  7. 2025Mitigating Hallucinations in Large Vision-Language Models with Internal Fact-based Contrastive Decoding
  8. 2025MINT: Mitigating Hallucinations in Large Vision-Language Models via Token Reduction