mlsanityData to 14 Oct 2026

The Mirage of Performance Gains: Why Contrastive Decoding Fails to Mitigate Object Hallucinations in MLLMs?

arXiv 2504.10020 ↗First posted 14 Apr 2025cs.CL cs.AI cs.CVcited by 9

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Contrastive decoding strategies are widely used to reduce object hallucinations in multimodal large language models (MLLMs). These methods work by constructing contrastive samples to induce hallucinations and then suppressing them in the output distribution. However, this paper demonstrates that such approaches fail to effectively mitigate the hallucination problem. The performance improvements observed on POPE Benchmark are largely driven by two misleading factors: (1) crude, unidirectional adjustments to the model's output distribution and (2) the adaptive plausibility constraint, which reduces the sampling strategy to greedy search. To further illustrate these issues, we introduce a series of spurious improvement methods and evaluate their performance against contrastive decoding techniques. Experimental results reveal that the observed performance gains in contrastive decoding are entirely unrelated to its intended goal of mitigating hallucinations. Our findings challenge common assumptions about the effectiveness of contrastive decoding strategies and pave the way for developing genuinely effective solutions to hallucinations in MLLMs.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 39 of its 45 references that are in the map. First posted on arXiv 14 Apr 2025.

Builds on

39 of 45 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024VaLiD: Mitigating the Hallucination of Large Vision Language Models by Visual Layer Fusion Contrastive Decoding24 Nov
  2. 2024Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Models4 Aug
  3. 2024Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding27 Mar
  4. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding1 Mar
  5. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  6. 2023Negative Object Presence Evaluation (NOPE) to Measure Object Hallucination in Vision-Language Models9 Oct
  7. 2023Detecting and Preventing Hallucinations in Large Vision Language Models11 Aug
  8. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun
  9. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 30

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  7. 2023Qwen Technical Reportcited by 3,344
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  10. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  11. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  12. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
18 more
  1. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  2. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  3. 2023A Survey on Multimodal Large Language Modelscited by 909
  4. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  5. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  6. 2018Object Hallucination in Image Captioningcited by 669
  7. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 609
  8. 2018nocaps: novel object captioning at scalecited by 444
  9. 2020Counterfactual VQA: A Cause-Effect Look at Language Biascited by 266
  10. 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Drivingcited by 232
  11. 2023ChatCAD: Interactive Computer-Aided Diagnosis on Medical Image using Large Language Modelscited by 116
  12. 2023Embodied Task Planning with Large Language Modelscited by 84
  13. 2022SwapMix: Diagnosing and Regularizing the Over-Reliance on Visual Context in Visual Question Answeringcited by 38
  14. 2023LLM as A Robotic Brain: Unifying Egocentric Memory and Controlcited by 33
  15. 2023Advancing Medical Imaging with Language Models: A Journey from N-grams to ChatGPTcited by 28
  16. 2022Visual Perturbation-aware Collaborative Learning for Overcoming the Language Prior Problemcited by 18
  17. 2023LLM-Based Human-Robot Collaboration Framework for Manipulation Taskscited by 17
  18. 2022Overcoming Language Priors in Visual Question Answering via Distinguishing Superficially Similar Instancescited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 7.

9 papers.

  1. 2025Cure or Poison? Embedding Instructions Visually Alters Hallucination in Vision-Language Modelscited by 4
  2. 2026TGIF: Text-Guided Layer Fusion Mitigates Hallucination in Multimodal LLMscited by 2
  3. 2026Do We Really Need External Tools to Mitigate Hallucinations? SIRA: Shared-Prefix Internal Reconstruction of Attributioncited by 1
  4. 2025Two Causes, Not One: Rethinking Omission and Fabrication Hallucinations in MLLMscited by 0
  5. 2026The Cost of Language: Centroid Erasure Exposes and Exploits Modal Competition in Multimodal Language Modelscited by 0
  6. 2026Rethinking CD: A Reproducibility Study and Extension on the Ineffectiveness of Contrastive Decoding at Mitigating Object Hallucinations in MLLMscited by 0
  7. 2026OPD-Aha: From Linguistic Momentum to Visual Reflection in Multimodal On-Policy Distillationcited by 0
  8. 2026Relevant Evidence Decoding for Audio-Visual Hallucination Mitigationcited by 0
  9. 2026Lost in Contrast: Instruction-Preserving Decoding for Reliable Visual Question Answeringcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024MLLM can see? Dynamic Correction Decoding for Hallucination Mitigation
  2. 2024Delve into Visual Contrastive Decoding for Hallucination Mitigation of Large Vision-Language Models
  3. 2024Mitigating Hallucinations in Large Vision-Language Models (LVLMs) via Language-Contrastive Decoding (LCD)
  4. 2025Mitigating Hallucinations in Large Vision-Language Models with Internal Fact-based Contrastive Decoding
  5. 2024ConVis: Contrastive Decoding with Hallucination Visualization for Mitigating Hallucinations in Multimodal Large Language Models
  6. 2026SDCD: Structure-Disrupted Contrastive Decoding for Mitigating Hallucinations in Large Vision-Language Models
  7. 2025MaskCD: Mitigating LVLM Hallucinations by Image Head Masked Contrastive Decoding
  8. 2025AVCD: Mitigating Hallucinations in Audio-Visual Large Language Models through Contrastive Decoding