mlsanityData to 14 Oct 2026

ConVis: Contrastive Decoding with Hallucination Visualization for Mitigating Hallucinations in Multimodal Large Language Models

arXiv 2408.13906 ↗First posted 25 Aug 2024cs.CV cs.AI cs.LGcited by 20

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Hallucinations in Multimodal Large Language Models (MLLMs) where generated responses fail to accurately reflect the given image pose a significant challenge to their reliability. To address this, we introduce ConVis, a novel training-free contrastive decoding method. ConVis leverages a text-to-image (T2I) generation model to semantically reconstruct the given image from hallucinated captions. By comparing the contrasting probability distributions produced by the original and reconstructed images, ConVis enables MLLMs to capture visual contrastive signals that penalize hallucination generation. Notably, this method operates purely within the decoding process, eliminating the need for additional data or model updates. Our extensive experiments on five popular benchmarks demonstrate that ConVis effectively reduces hallucinations across various MLLMs, highlighting its potential to enhance model reliability.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 16 of its 24 references that are in the map. First posted on arXiv 25 Aug 2024.

Builds on

16 of 24 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding1 Mar
  2. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback1 Dec
  3. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  4. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  5. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun

Other references · 11

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  5. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  6. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  7. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  8. 2018Object Hallucination in Image Captioningcited by 669
  9. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 609
  10. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 547
  11. 2024Hyper-SD: Trajectory Segmented Consistency Model for Efficient Image Synthesiscited by 156

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 8; 2026: 11.

20 papers.

Sort:

By area:

  1. 2024Look Twice Before You Answer: Memory-Space Visual Retracing for Hallucination Mitigation in Multimodal Large Language Modelscited by 72
  2. 2025Octopus: Alleviating Hallucination via Dynamic Contrastive Decodingcited by 24
  3. 2025Exploring Hallucination of Large Multimodal Models in Video Understanding: Benchmark, Analysis and Mitigationcited by 20
  4. 2025INTER: Mitigating Hallucination in Large Vision-Language Models by Interaction Guidance Samplingcited by 10
  5. 2025Policy Contrastive Decoding for Robotic Foundation Modelscited by 8
  6. 2026MoD-DPO: Towards Mitigating Cross-modal Hallucinations in Omni LLMs using Modality Decoupled Preference Optimizationcited by 6
  7. 2025UniVRSE: Unified Vision-conditioned Response Semantic Entropy for Hallucination Detection in Medical Vision-Language Modelscited by 5
  8. 2025Decoupling Contrastive Decoding: Robust Hallucination Mitigation in Multimodal Large Language Modelscited by 5
  9. 2026KVSmooth: Mitigating Hallucination in Multi-modal Large Language Models through Key-Value Smoothingcited by 5
  10. 20263D-VCD: Hallucination Mitigation in 3D-LLM Embodied Agents through Visual Contrastive Decodingcited by 3
  11. 2025Mitigating Multimodal Hallucinations via Gradient-based Self-Reflectioncited by 2
  12. 2026Linking Perception, Confidence and Accuracy in MLLMscited by 2
8 more
  1. 2025Self-Aug: Query and Entropy Adaptive Decoding for Large Vision-Language Modelscited by 1
  2. 2026BUZZY: Contrastive Scoring to Mitigate Text-Induced Bias in Multimodal Multiple-Choice QAcited by 1
  3. 2026Cross-modal Identity Mapping: Minimizing Information Loss in Modality Conversion via Reinforcement Learningcited by 0
  4. 2026Test-Time Hallucination Control in Large Vision-Language Modelscited by 0
  5. 2026Beyond Tokens: A Survey on Decoding Methods for Large Language and Vision-Language Modelscited by 0
  6. 2026Hallucination Mitigation for Large Vision-Language Models via Implicit Feature Stabilizationcited by 0
  7. 2026MLLMs Hallucinate when Information Distribution Drifts in Synergy Headscited by 0
  8. 2026Semantic-Spatial Agreement Verification for Mitigating Object Hallucination in Multimodal Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Delve into Visual Contrastive Decoding for Hallucination Mitigation of Large Vision-Language Models
  2. 2025The Mirage of Performance Gains: Why Contrastive Decoding Fails to Mitigate Object Hallucinations in MLLMs?
  3. 2026SDCD: Structure-Disrupted Contrastive Decoding for Mitigating Hallucinations in Large Vision-Language Models
  4. 2024CODE: Contrasting Self-generated Description to Combat Hallucination in Large Multi-modal Models
  5. 2024Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding
  6. 2025Mitigating Hallucinations in Large Vision-Language Models with Internal Fact-based Contrastive Decoding
  7. 2025AVCD: Mitigating Hallucinations in Audio-Visual Large Language Models through Contrastive Decoding
  8. 2025Mitigating Hallucination in Multimodal LLMs with Layer Contrastive Decoding