# ConVis: Contrastive Decoding with Hallucination Visualization for Mitigating Hallucinations in Multimodal Large Language Models

arXiv: https://arxiv.org/abs/2408.13906 · HTML: https://mlsanity.com/paper/2408.13906/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 16 of its 24 references that are in the map.
- First posted on arXiv 25 Aug 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Hallucination mitigation in LVLMs](https://mlsanity.com/mllm/hallucination-mitigation-in-lvlms/)

## Builds on (most cited first, 16 of 16)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding](https://mlsanity.com/paper/2311.16922/) (2023)
- [Object Hallucination in Image Captioning](https://mlsanity.com/paper/1809.02156/) (2018)
- [Contrastive Decoding: Open-ended Text Generation as Optimization](https://mlsanity.com/paper/2210.15097/) (2022)
- [mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration](https://mlsanity.com/paper/2311.04257/) (2023)
- [OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation](https://mlsanity.com/paper/2311.17911/) (2023)
- [Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning](https://mlsanity.com/paper/2306.14565/) (2023)
- [RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback](https://mlsanity.com/paper/2312.00849/) (2023)
- [HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding](https://mlsanity.com/paper/2403.00425/) (2024)
- [Hyper-SD: Trajectory Segmented Consistency Model for Efficient Image Synthesis](https://mlsanity.com/paper/2404.13686/) (2024)

## Built on by (most cited first, 20 of 20)

- [Look Twice Before You Answer: Memory-Space Visual Retracing for Hallucination Mitigation in Multimodal Large Language Models](https://mlsanity.com/paper/2410.03577/) (2024)
- [Octopus: Alleviating Hallucination via Dynamic Contrastive Decoding](https://mlsanity.com/paper/2503.00361/) (2025)
- [Exploring Hallucination of Large Multimodal Models in Video Understanding: Benchmark, Analysis and Mitigation](https://mlsanity.com/paper/2503.19622/) (2025)
- [INTER: Mitigating Hallucination in Large Vision-Language Models by Interaction Guidance Sampling](https://mlsanity.com/paper/2507.05056/) (2025)
- [Policy Contrastive Decoding for Robotic Foundation Models](https://mlsanity.com/paper/2505.13255/) (2025)
- [MoD-DPO: Towards Mitigating Cross-modal Hallucinations in Omni LLMs using Modality Decoupled Preference Optimization](https://mlsanity.com/paper/2603.03192/) (2026)
- [UniVRSE: Unified Vision-conditioned Response Semantic Entropy for Hallucination Detection in Medical Vision-Language Models](https://mlsanity.com/paper/2503.20504/) (2025)
- [Decoupling Contrastive Decoding: Robust Hallucination Mitigation in Multimodal Large Language Models](https://mlsanity.com/paper/2504.08809/) (2025)
- [KVSmooth: Mitigating Hallucination in Multi-modal Large Language Models through Key-Value Smoothing](https://mlsanity.com/paper/2602.04268/) (2026)
- [3D-VCD: Hallucination Mitigation in 3D-LLM Embodied Agents through Visual Contrastive Decoding](https://mlsanity.com/paper/2604.08645/) (2026)
- [Mitigating Multimodal Hallucinations via Gradient-based Self-Reflection](https://mlsanity.com/paper/2509.03113/) (2025)
- [Linking Perception, Confidence and Accuracy in MLLMs](https://mlsanity.com/paper/2603.12149/) (2026)
- [Self-Aug: Query and Entropy Adaptive Decoding for Large Vision-Language Models](https://mlsanity.com/paper/2510.13315/) (2025)
- [BUZZY: Contrastive Scoring to Mitigate Text-Induced Bias in Multimodal Multiple-Choice QA](https://mlsanity.com/paper/2603.28026/) (2026)
- [Cross-modal Identity Mapping: Minimizing Information Loss in Modality Conversion via Reinforcement Learning](https://mlsanity.com/paper/2603.01696/) (2026)
- [Test-Time Hallucination Control in Large Vision-Language Models](https://mlsanity.com/paper/2608.11474/) (2026)
- [Beyond Tokens: A Survey on Decoding Methods for Large Language and Vision-Language Models](https://mlsanity.com/paper/2608.14797/) (2026)
- [Hallucination Mitigation for Large Vision-Language Models via Implicit Feature Stabilization](https://mlsanity.com/paper/2608.29924/) (2026)
- [MLLMs Hallucinate when Information Distribution Drifts in Synergy Heads](https://mlsanity.com/paper/2609.09206/) (2026)
- [Semantic-Spatial Agreement Verification for Mitigating Object Hallucination in Multimodal Large Language Models](https://mlsanity.com/paper/2609.17269/) (2026)

## Abstract (arXiv)

Hallucinations in Multimodal Large Language Models (MLLMs) where generated responses fail to accurately reflect the given image pose a significant challenge to their reliability. To address this, we introduce ConVis, a novel training-free contrastive decoding method. ConVis leverages a text-to-image (T2I) generation model to semantically reconstruct the given image from hallucinated captions. By comparing the contrasting probability distributions produced by the original and reconstructed images, ConVis enables MLLMs to capture visual contrastive signals that penalize hallucination generation. Notably, this method operates purely within the decoding process, eliminating the need for additional data or model updates. Our extensive experiments on five popular benchmarks demonstrate that ConVis effectively reduces hallucinations across various MLLMs, highlighting its potential to enhance model reliability.