# Mitigating Hallucinations in Large Vision-Language Models (LVLMs) via Language-Contrastive Decoding (LCD)

arXiv: https://arxiv.org/abs/2408.04664 · HTML: https://mlsanity.com/paper/2408.04664/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 11 of its 24 references that are in the map.
- First posted on arXiv 6 Aug 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Hallucination mitigation in LVLMs](https://mlsanity.com/mllm/hallucination-mitigation-in-lvlms/)

## Builds on (most cited first, 11 of 11)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding](https://mlsanity.com/paper/2311.16922/) (2023)
- [Contrastive Decoding: Open-ended Text Generation as Optimization](https://mlsanity.com/paper/2210.15097/) (2022)
- [Multi-Modal Hallucination Control by Visual Information Grounding](https://mlsanity.com/paper/2403.14003/) (2024)
- [Trading Off Diversity and Quality in Natural Language Generation](https://mlsanity.com/paper/2004.10450/) (2020)
- [Mitigating Fine-Grained Hallucination by Fine-Tuning Large Vision-Language Models with Caption Rewrites](https://mlsanity.com/paper/2312.01701/) (2023)
- [Mitigating Hallucinations and Off-target Machine Translation with Source-Contrastive and Language-Contrastive Decoding](https://mlsanity.com/paper/2309.07098/) (2023)

## Built on by (most cited first, 23 of 23)

- [Octopus: Alleviating Hallucination via Dynamic Contrastive Decoding](https://mlsanity.com/paper/2503.00361/) (2025)
- [Mitigating Hallucinations in Large Vision-Language Models via Summary-Guided Decoding](https://mlsanity.com/paper/2410.13321/) (2024)
- [G2: Guided Generation for Enhanced Output Diversity in LLMs](https://mlsanity.com/paper/2511.00432/) (2025)
- [Med-VCD: Mitigating Hallucination for Medical Large Vision Language Models through Visual Contrastive Decoding](https://mlsanity.com/paper/2512.01922/) (2025)
- [VIBE: Can a VLM Read the Room?](https://mlsanity.com/paper/2506.11162/) (2025)
- [Reallocating Attention Across Layers to Reduce Multimodal Hallucination](https://mlsanity.com/paper/2510.10285/) (2025)
- [Reflective Preference Optimization (RPO): Enhancing On-Policy Alignment via Hint-Guided Reflection](https://mlsanity.com/paper/2512.13240/) (2025)
- [SDCD: Structure-Disrupted Contrastive Decoding for Mitigating Hallucinations in Large Vision-Language Models](https://mlsanity.com/paper/2601.03500/) (2026)
- [Temporal Contrastive Decoding: A Training-Free Method for Large Audio-Language Models](https://mlsanity.com/paper/2604.15383/) (2026)
- [FADE: Mitigating Hallucinations by Reducing Language-Prior Dominance in Large Vision-Language Models](https://mlsanity.com/paper/2606.29431/) (2026)
- [Do Vision Encoders Truly Explain Object Hallucination?: Mitigating Object Hallucination via Simple Fine-Grained CLIPScore](https://mlsanity.com/paper/2502.20034/) (2025)
- [Mitigating Multimodal Hallucinations via Gradient-based Self-Reflection](https://mlsanity.com/paper/2509.03113/) (2025)
- [Toward More Reliable Artificial Intelligence: Reducing Hallucinations in Vision-Language Models](https://mlsanity.com/paper/2512.07564/) (2025)
- [YARD: Y-Architecture Register Decoding for Efficient Hallucination Mitigation in Large Vision-Language Models](https://mlsanity.com/paper/2605.31429/) (2026)
- [HalCECE: A Framework for Explainable Hallucination Detection through Conceptual Counterfactuals in Image Captioning](https://mlsanity.com/paper/2503.00436/) (2025)
- [Rationale-Enhanced Decoding for Multi-modal Chain-of-Thought](https://mlsanity.com/paper/2507.07685/) (2025)
- [Self-Consistency as a Free Lunch: Reducing Hallucinations in Vision-Language Models via Self-Reflection](https://mlsanity.com/paper/2509.23236/) (2025)
- [Self-Correction Inside the Model: Leveraging Layer Attention to Mitigate Hallucinations in Large Vision Language Models](https://mlsanity.com/paper/2603.00437/) (2026)
- [Segmentation-Based Attention Entropy: Detecting and Mitigating Object Hallucinations in Large Vision-Language Models](https://mlsanity.com/paper/2603.16558/) (2026)
- [Hallucination-aware intermediate representation edit in large vision-language models](https://mlsanity.com/paper/2603.29405/) (2026)
- [HTDC: Hesitation-Triggered Differential Calibration for Mitigating Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2604.12115/) (2026)
- [The Cost of Language: Centroid Erasure Exposes and Exploits Modal Competition in Multimodal Language Models](https://mlsanity.com/paper/2604.14363/) (2026)
- [Learning from Fine-Grained Visual Discrepancies: Mitigating Multimodal Hallucinations via In-Context Visual Contrastive Optimization](https://mlsanity.com/paper/2605.31312/) (2026)

## Abstract (arXiv)

Large Vision-Language Models (LVLMs) are an extension of Large Language Models (LLMs) that facilitate processing both image and text inputs, expanding AI capabilities. However, LVLMs struggle with object hallucinations due to their reliance on text cues and learned object co-occurrence biases. While most research quantifies these hallucinations, mitigation strategies are still lacking. Our study introduces a Language Contrastive Decoding (LCD) algorithm that adjusts LVLM outputs based on LLM distribution confidence levels, effectively reducing object hallucinations. We demonstrate the advantages of LCD in leading LVLMs, showing up to %4 improvement in POPE F1 scores and up to %36 reduction in CHAIR scores on the COCO validation set, while also improving captioning quality scores. Our method effectively improves LVLMs without needing complex post-processing or retraining, and is easily applicable to different models. Our findings highlight the potential of further exploration of LVLM-specific decoding algorithms.