mlsanityData to 14 Oct 2026

Grounding Language with Vision: A Conditional Mutual Information Calibrated Decoding Strategy for Reducing Hallucinations in LVLMs

arXiv 2505.19678 ↗First posted 26 May 2025cs.CL cs.CVcited by 17

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Large Vision-Language Models (LVLMs) are susceptible to hallucinations, where generated responses seem semantically plausible yet exhibit little or no relevance to the input image. Previous studies reveal that this issue primarily stems from LVLMs' over-reliance on language priors while disregarding the visual information during decoding. To alleviate this issue, we introduce a novel Conditional Pointwise Mutual Information (C-PMI) calibrated decoding strategy, which adaptively strengthens the mutual dependency between generated texts and input images to mitigate hallucinations. Unlike existing methods solely focusing on text token sampling, we propose to jointly model the contributions of visual and textual tokens to C-PMI, formulating hallucination mitigation as a bi-level optimization problem aimed at maximizing mutual information. To solve it, we design a token purification mechanism that dynamically regulates the decoding process by sampling text tokens remaining maximally relevant to the given image, while simultaneously refining image tokens most pertinent to the generated response. Extensive experiments across various benchmarks reveal that the proposed method significantly reduces hallucinations in LVLMs while preserving decoding efficiency.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 43 of its 51 references that are in the map. First posted on arXiv 26 May 2025.

Builds on

43 of 51 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024MLLM can see? Dynamic Correction Decoding for Hallucination Mitigation15 Oct
  2. 2024Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Models4 Aug
  3. 2024Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding27 Mar
  4. 2024Multi-Modal Hallucination Control by Visual Information Grounding20 Mar
  5. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding1 Mar
  6. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback1 Dec
  7. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  8. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  9. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization28 Nov
  10. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Models24 Oct
  11. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Models1 Oct
  12. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun
1 more
  1. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 30

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2023Visual Instruction Tuningcited by 7,869
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  9. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  10. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  11. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  12. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
18 more
  1. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  2. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  3. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  4. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 864
  5. 2018Object Hallucination in Image Captioningcited by 669
  6. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 642
  7. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 376
  8. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 290
  9. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 267
  10. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Drivingcited by 233
  11. 2024Inducing High Energy-Latency of Large Vision-Language Models with Verbose Imagescited by 69
  12. 2022Mutual Information Alleviates Hallucinations in Abstractive Summarizationcited by 68
  13. 2024One Perturbation is Enough: On Generating Universal Adversarial Perturbations against Vision-Language Pre-training Modelscited by 40
  14. 2024Open-FinLLMs: Open Multimodal Large Language Models for Financial Applicationscited by 36
  15. 2024Benchmarking Open-ended Audio Dialogue Understanding for Large Audio-Language Modelscited by 29
  16. 2024CLIP-Guided Generative Networks for Transferable Targeted Adversarial Attackscited by 19
  17. 2025Your Language Model Can Secretly Write Like Humans: Contrastive Paraphrase Attacks on LLM-Generated Text Detectorscited by 10
  18. 2022REQA: Coarse-to-fine Assessment of Image Quality to Alleviate the Range Effectcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 16.

17 papers.

Sort:

By area:

  1. 2025HLFormer: Enhancing Partially Relevant Video Retrieval with Hyperbolic Learningcited by 17
  2. 2026Hallucination Begins Where Saliency Dropscited by 9
  3. 2026Seeing Through the Chain: Mitigate Hallucination in Multimodal Reasoning Models via CoT Compression and Contrastive Preference Optimizationcited by 5
  4. 2026Improving Visual Reasoning with Iterative Evidence Refinementcited by 5
  5. 2026Decoding by Perturbation: Mitigating MLLM Hallucinations via Dynamic Textual Perturbationcited by 3
  6. 2026Heterogeneous Uncertainty-Guided Composed Image Retrieval with Fine-Grained Probabilistic Learningcited by 2
  7. 2026SAVAA: Mitigating Hallucinations in LVLMs via Step-wise Adaptive Visual Attention Amplificationcited by 2
  8. 2026AdaIAT: Adaptively Increasing Attention to Generated Text to Alleviate Hallucinations in LVLMcited by 2
  9. 2026Looking Back and Forth: Cross-Image Attention Calibration and Attentive Preference Learning for Multi-Image Hallucination Mitigationcited by 2
  10. 2026One Token, Two Fates: A Unified Framework via Vision Token Manipulation Against MLLMs Hallucinationcited by 2
  11. 2026SAGE: Sink-Aware Grounded Decoding for Multimodal Hallucination Mitigationcited by 2
  12. 2026SAKED: Mitigating Hallucination in Large Vision-Language Models via Stability-Aware Knowledge Enhanced Decodingcited by 1
5 more
  1. 2026Steer Where It Matters: Token-Level Visual-Sensitivity Steering for LVLMs Hallucination Mitigationcited by 1
  2. 2026MIRL: Mutual Information-Guided Reinforcement Learning for Vision-Language Modelscited by 0
  3. 2026MergeHEIR: Mitigating Multimodal Hallucinations as the Tax of Model Mergingcited by 0
  4. 2026DARE to Mitigate Hallucination: Dual-path Auto-Regressive-aware Editingcited by 0
  5. 2026Relevant Evidence Decoding for Audio-Visual Hallucination Mitigationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Mixture of Decoding: An Attention-Inspired Adaptive Decoding Strategy to Mitigate Hallucinations in Large Vision-Language Models
  2. 2024Reducing Hallucinations in Vision-Language Models via Latent Space Steering
  3. 2025MINT: Mitigating Hallucinations in Large Vision-Language Models via Token Reduction
  4. 2024Mitigating Hallucinations in Large Vision-Language Models via Summary-Guided Decoding
  5. 2025ONLY: One-Layer Intervention Sufficiently Mitigates Hallucinations in Large Vision-Language Models
  6. 2025Self-Correcting Decoding with Generative Feedback for Mitigating Hallucinations in Large Vision-Language Models
  7. 2026CHASD: Language Increment-Calibrated Contrastive Decoding against Hallucination in LVLMs
  8. 2026Context-Aware Decoding for Faithful Vision-Language Generation