mlsanityData to 7 Oct 2026

Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding

arXiv 2311.16922 ↗First posted 28 Nov 2023cs.CV cs.AI cs.CLcited by 671

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Large Vision-Language Models (LVLMs) have advanced considerably, intertwining visual recognition and language understanding to generate content that is not only coherent but also contextually attuned. Despite their success, LVLMs still suffer from the issue of object hallucinations, where models generate plausible yet incorrect outputs that include objects that do not exist in the images. To mitigate this issue, we introduce Visual Contrastive Decoding (VCD), a simple and training-free method that contrasts output distributions derived from original and distorted visual inputs. The proposed VCD effectively reduces the over-reliance on statistical bias and unimodal priors, two essential causes of object hallucinations. This adjustment ensures the generated content is closely grounded to visual inputs, resulting in contextually accurate outputs. Our experiments show that VCD, without either additional training or the usage of external tools, significantly mitigates the object hallucination issue across different LVLM families. Beyond mitigating object hallucinations, VCD also excels in general LVLM benchmarks, highlighting its wide-ranging applicability.

Key facts. Cited by 671 papers in the map (data through 7 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 69 of its 81 references that are in the map. First posted on arXiv 28 Nov 2023.

Builds on

69 of 81 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Models24 Oct
  2. 2023Negative Object Presence Evaluation (NOPE) to Measure Object Hallucination in Vision-Language Models9 Oct
  3. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Models1 Oct
  4. 2023Detecting and Preventing Hallucinations in Large Vision Language Models11 Aug
  5. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun
  6. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 63

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2023Visual Instruction Tuningcited by 7,792
  9. 2020Scaling Laws for Neural Language Modelscited by 6,411
  10. 2022Classifier-Free Diffusion Guidancecited by 5,717
  11. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  12. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
38 more
  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  3. 2023Qwen Technical Reportcited by 3,273
  4. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  5. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  6. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  7. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  8. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  9. 2022Emergent Abilities of Large Language Modelscited by 2,483
  10. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  11. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  12. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  13. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  14. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  15. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  16. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  17. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 849
  18. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 786
  19. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  20. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  21. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  22. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  23. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  24. 2018Object Hallucination in Image Captioningcited by 661
  25. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 598
  26. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  27. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  28. 2021DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Expertscited by 434
  29. 2023Trusting Your Evidence: Hallucinate Less with Context-aware Decodingcited by 319
  30. 2022UL2: Unifying Language Learning Paradigmscited by 292
  31. 2020Counterfactual VQA: A Cause-Effect Look at Language Biascited by 266
  32. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Conceptscited by 255
  33. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  34. 2017Beam Search Strategies for Neural Machine Translationcited by 232
  35. 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Drivingcited by 231
  36. 2020Detecting Hallucinated Content in Conditional Neural Sequence Generationcited by 151
  37. 2023M3IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuningcited by 126
  38. 2016Analyzing the Behavior of Visual Question Answering Modelscited by 119

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 126; 2025: 233; 2026: 312.

671 papers.

Sort:

By area:

  1. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 380
  2. 2024A Survey on Hallucination in Large Vision-Language Modelscited by 352
  3. 2024Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decodingcited by 198
  4. 2024Aligning Modalities in Vision Large Language Models via Preference Fine-tuningcited by 197
  5. 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMscited by 174
  6. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decodingcited by 171
  7. 2024Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lenscited by 139
  8. 2024RLAIF-V: Open-Source AI Feedback Leads to Super GPT-4V Trustworthinesscited by 129
  9. 2024The Emerged Security and Privacy of LLM Agent: A Survey with Case Studiescited by 127
  10. 2025See What You Are Told: Visual Attention Sink in Large Multimodal Modelscited by 127
  11. 2024IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decodingcited by 118
  12. 2024A Comprehensive Survey of Hallucination in Large Language, Image, Video and Audio Foundation Modelscited by 98
38 more
  1. 2024Mementos: A Comprehensive Benchmark for Multimodal Large Language Model Reasoning over Image Sequencescited by 93
  2. 2025Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective on Focus Areascited by 93
  3. 2024Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Modelscited by 90
  4. 2024Tuning Language Models by Proxycited by 88
  5. 2024MLLM can see? Dynamic Correction Decoding for Hallucination Mitigationcited by 87
  6. 2024MMed-RAG: Versatile Multimodal RAG System for Medical Vision Language Modelscited by 84
  7. 2024Calibrated Self-Rewarding Vision Language Modelscited by 77
  8. 2024Unified Hallucination Detection for Multimodal Large Language Modelscited by 75
  9. 2024Mitigating Object Hallucinations in Large Vision-Language Models with Assembly of Global and Local Attentioncited by 74
  10. 2024Look Twice Before You Answer: Memory-Space Visual Retracing for Hallucination Mitigation in Multimodal Large Language Modelscited by 72
  11. 2024VidHalluc: Evaluating Temporal Hallucinations in Multimodal Large Language Models for Video Understandingcited by 65
  12. 2024Factuality of Large Language Models: A Surveycited by 64
  13. 2025Mitigating Hallucinations in Large Vision-Language Models via DPO: On-Policy Data Hold the Keycited by 64
  14. 2024Contrastive Region Guidance: Improving Grounding in Vision-Language Models without Trainingcited by 62
  15. 2025The Hidden Life of Tokens: Reducing Hallucination of Large Vision-Language Models via Visual Information Steeringcited by 62
  16. 2024RULE: Reliable Multimodal RAG for Factuality in Medical Vision Language Modelscited by 58
  17. 2024Reducing Hallucinations in Vision-Language Models via Latent Space Steeringcited by 56
  18. 2024EventHallusion: Diagnosing Event Hallucinations in Video LLMscited by 55
  19. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  20. 2024TinyChart: Efficient Chart Understanding with Visual Token Merging and Program-of-Thoughts Learningcited by 51
  21. 2024MLLM Is a Strong Reranker: Advancing Multimodal Retrieval-augmented Generation via Knowledge-enhanced Reranking and Noise-injected Trainingcited by 50
  22. 2025LLM-based Agents Suffer from Hallucinations: A Survey of Taxonomy, Methods, and Directionscited by 49
  23. 2024The Curse of Multi-Modalities: Evaluating Hallucinations of Large Multimodal Models across Language, Visual, and Audiocited by 48
  24. 2024Mitigating Object Hallucination via Concentric Causal Attentioncited by 48
  25. 2024Logical Closed Loop: Uncovering Object Hallucinations in Large Vision-Language Modelscited by 47
  26. 2024How Easy is It to Fool Your Multimodal LLMs? An Empirical Analysis on Deceptive Promptscited by 47
  27. 2024DAMRO: Dive into the Attention Mechanism of LVLM to Reduce Object Hallucinationcited by 47
  28. 2024Don't Miss the Forest for the Trees: Attentional Vision Calibration for Large Vision Language Modelscited by 46
  29. 2024Nullu: Mitigating Object Hallucinations in Large Vision-Language Models via HalluSpace Projectioncited by 46
  30. 2024Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidancecited by 44
  31. 2024PhD: A ChatGPT-Prompted Visual hallucination Evaluation Datasetcited by 44
  32. 2025VARGPT: Unified Understanding and Generation in a Visual Autoregressive Multimodal Large Language Modelcited by 44
  33. 2025Self-Correcting Decoding with Generative Feedback for Mitigating Hallucinations in Large Vision-Language Modelscited by 44
  34. 2024LLaVA-MoD: Making LLaVA Tiny via MoE Knowledge Distillationcited by 43
  35. 2025Seeing Far and Clearly: Mitigating Hallucinations in MLLMs with Attention Causal Decodingcited by 42
  36. 2025ClearSight: Visual Signal Enhancement for Object Hallucination Mitigation in Multimodal Large language Modelscited by 40
  37. 2024CODE: Contrasting Self-generated Description to Combat Hallucination in Large Multi-modal Modelscited by 37
  38. 2024Insight Over Sight: Exploring the Vision-Knowledge Conflicts in Multimodal LLMscited by 37

50 most cited of 671 All 671 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025SECOND: Mitigating Perceptual Hallucination in Vision-Language Models via Selective and Contrastive Decoding
  2. 2024Delve into Visual Contrastive Decoding for Hallucination Mitigation of Large Vision-Language Models
  3. 2025Mitigating Hallucinations in Large Vision-Language Models with Internal Fact-based Contrastive Decoding
  4. 2024From Pixels to Tokens: Revisiting Object Hallucinations in Large Vision-Language Models
  5. 2022Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-training
  6. 2026CHASD: Language Increment-Calibrated Contrastive Decoding against Hallucination in LVLMs
  7. 2025MINT: Mitigating Hallucinations in Large Vision-Language Models via Token Reduction
  8. 2024Interpreting and Editing Vision-Language Representations to Mitigate Hallucinations