mlsanityData to 14 Oct 2026

CODE: Contrasting Self-generated Description to Combat Hallucination in Large Multi-modal Models

arXiv 2406.01920 ↗First posted 4 Jun 2024cs.CV cs.AIcited by 38

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Large Multi-modal Models (LMMs) have recently demonstrated remarkable abilities in visual context understanding and coherent response generation. However, alongside these advancements, the issue of hallucinations has emerged as a significant challenge, producing erroneous responses that are unrelated to the visual contents. In this paper, we introduce a novel contrastive-based decoding method, COuntering DEscription Contrastive Decoding (CODE), which leverages self-generated descriptions as contrasting references during the decoding phase of LMMs to address hallucination issues. CODE utilizes the comprehensive descriptions from model itself as visual counterpart to correct and improve response alignment with actual visual content. By dynamically adjusting the information flow and distribution of next-token predictions in the LMM's vocabulary, CODE enhances the coherence and informativeness of generated responses. Extensive experiments demonstrate that our method significantly reduces hallucinations and improves cross-modal consistency across various benchmarks and cutting-edge LMMs. Our method provides a simple yet effective decoding strategy that can be integrated to existing LMM frameworks without additional training.

Key facts. Cited by 38 papers in the map (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 58 of its 64 references that are in the map. First posted on arXiv 4 Jun 2024.

Builds on

58 of 64 references.

Closest ancestors · 14

Same direction, newest first

  1. 2024Hallucination of Multimodal Large Language Models: A Survey29 Apr
  2. 2024What if...?: Thinking Counterfactual Keywords Helps to Mitigate Hallucination in Large Multi-modal Models20 Mar
  3. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding1 Mar
  4. 2024IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decoding28 Feb
  5. 2024A Survey on Hallucination in Large Vision-Language Models1 Feb
  6. 2023Mitigating Fine-Grained Hallucination by Fine-Tuning Large Vision-Language Models with Caption Rewrites4 Dec
  7. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback1 Dec
  8. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  9. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  10. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization28 Nov
  11. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation13 Nov
  12. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Models1 Oct
2 more
  1. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun
  2. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 44

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  10. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  11. 2015Fast R-CNNcited by 2,933
  12. 2019The Curious Case of Neural Text Degenerationcited by 2,916
32 more
  1. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  2. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  3. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  4. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  5. 2015VQA: Visual Question Answeringcited by 2,506
  6. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  7. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  8. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,423
  9. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,312
  10. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  11. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  12. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  13. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 735
  14. 2024Yi: Open Foundation Models by 01.AIcited by 719
  15. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  16. 2018Object Hallucination in Image Captioningcited by 669
  17. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 609
  18. 2018The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scalecited by 544
  19. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  20. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  21. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 421
  22. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 376
  23. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 365
  24. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  25. 2023Chain-of-Verification Reduces Hallucination in Large Language Modelscited by 324
  26. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 300
  27. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 232
  28. 2023Honeybee: Locality-enhanced Projector for Multimodal LLMcited by 197
  29. 2021UniT: Multimodal Multitask Learning with a Unified Transformercited by 195
  30. 2021UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modelingcited by 122
  31. 2024CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Expertscited by 56
  32. 2024MoAI: Mixture of All Intelligence for Large Language and Vision Modelscited by 36

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 21; 2026: 14.

38 papers.

Sort:

By area:

  1. 2025Self-Correcting Decoding with Generative Feedback for Mitigating Hallucinations in Large Vision-Language Modelscited by 44
  2. 2024ICT: Image-Object Cross-Level Trusted Intervention for Mitigating Object Hallucination in Large Vision-Language Modelscited by 30
  3. 2025AVCD: Mitigating Hallucinations in Audio-Visual Large Language Models through Contrastive Decodingcited by 30
  4. 2025Large Vision-Language Model Alignment and Misalignment: A Survey Through the Lens of Explainabilitycited by 28
  5. 2025Exploring Hallucination of Large Multimodal Models in Video Understanding: Benchmark, Analysis and Mitigationcited by 20
  6. 2024Cracking the Code of Hallucination in LVLMs with Vision-aware Head Divergencecited by 19
  7. 2025SECOND: Mitigating Perceptual Hallucination in Vision-Language Models via Selective and Contrastive Decodingcited by 19
  8. 2025TruthPrInt: Mitigating Large Vision-Language Models Object Hallucination Via Latent Truthful-Guided Pre-Interventioncited by 11
  9. 2025Why LVLMs Are More Prone to Hallucinations in Longer Responses: The Role of Contextcited by 10
  10. 2025When Continue Learning Meets Multimodal Large Language Model: A Surveycited by 9
  11. 2025OmniDPO: A Preference Optimization Framework to Address Omni-Modal Hallucinationcited by 9
  12. 2025Fork-Merge Decoding: Enhancing Multimodal Understanding in Audio-Visual Large Language Modelscited by 8
26 more
  1. 2025Revealing Perception and Generation Dynamics in LVLMs: Mitigating Hallucinations via Validated Dominance Correctioncited by 7
  2. 2026On the Nature of Attention Sink that Shapes Decoding Strategy in Omni-LLMscited by 6
  3. 2024DDFAV: Remote Sensing Large Vision Language Models Dataset and Evaluation Benchmarkcited by 5
  4. 2025Image Tokens Matter: Mitigating Hallucination in Discrete Tokenizer-based Large Vision-Language Models via Latent Editingcited by 5
  5. 2026MAD: Modality-Adaptive Decoding for Mitigating Cross-Modal Hallucinations in Multimodal Large Language Modelscited by 5
  6. 2025Revisit What You See: Revealing Visual Semantics in Vision Tokens to Guide LVLM Decodingcited by 4
  7. 2025HalluRNN: Mitigating Hallucinations via Recurrent Cross-Layer Reasoning in Large Vision-Language Modelscited by 4
  8. 2026CRoPS: A Training-Free Hallucination Mitigation Framework for Vision-Language Modelscited by 4
  9. 2026Revealing and Enhancing Core Visual Regions: Harnessing Internal Attention Dynamics for Hallucination Mitigation in LVLMscited by 4
  10. 2026Seeing Farther and Smarter: Value-Guided Multi-Path Reflection for VLM Policy Optimizationcited by 4
  11. 2026Mitigating Manifold Departure: Uncertainty-Aware Subspace Rectification for Trustworthy MLLM Decodingcited by 4
  12. 2025EyePCR: A Comprehensive Benchmark for Fine-Grained Perception, Knowledge Comprehension and Clinical Reasoning in Ophthalmic Surgerycited by 3
  13. 2026Mask What Matters: Mitigating Object Hallucinations in Multimodal Large Language Models with Object-Aligned Visual Contrastive Decodingcited by 3
  14. 2026FADE: Mitigating Hallucinations by Reducing Language-Prior Dominance in Large Vision-Language Modelscited by 3
  15. 2025Preemptive Hallucination Reduction: An Input-Level Approach for Multimodal Language Modelcited by 2
  16. 2025Self-Aug: Query and Entropy Adaptive Decoding for Large Vision-Language Modelscited by 1
  17. 2026GEASS: Gated Evidence-Adaptive Selective Caption Trust for Vision-Language Modelscited by 1
  18. 2026Do We Really Need External Tools to Mitigate Hallucinations? SIRA: Shared-Prefix Internal Reconstruction of Attributioncited by 1
  19. 2025Mitigating Object Hallucination via Robust Local Perception Searchcited by 0
  20. 2025Rationale-Enhanced Decoding for Multi-modal Chain-of-Thoughtcited by 0
  21. 2025TARS: MinMax Token-Adaptive Preference Strategy for Hallucination Reduction in MLLMscited by 0
  22. 2025Self-Consistency as a Free Lunch: Reducing Hallucinations in Vision-Language Models via Self-Reflectioncited by 0
  23. 2026HTDC: Hesitation-Triggered Differential Calibration for Mitigating Hallucination in Large Vision-Language Modelscited by 0
  24. 2026Rethinking Visual Neglect: Steering via Context-Preference for MLLM Hallucination Mitigationcited by 0
  25. 2026ReWEIGH the Evidence: Calibrating Token-Level Ordinal Visual Evidence to Mitigate Hallucinations in Large Vision-Language Modelscited by 0
  26. 2026Relevant Evidence Decoding for Audio-Visual Hallucination Mitigationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024ConVis: Contrastive Decoding with Hallucination Visualization for Mitigating Hallucinations in Multimodal Large Language Models
  2. 2024Delve into Visual Contrastive Decoding for Hallucination Mitigation of Large Vision-Language Models
  3. 2024Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding
  4. 2025ONLY: One-Layer Intervention Sufficiently Mitigates Hallucinations in Large Vision-Language Models
  5. 2026SDCD: Structure-Disrupted Contrastive Decoding for Mitigating Hallucinations in Large Vision-Language Models
  6. 2025Octopus: Alleviating Hallucination via Dynamic Contrastive Decoding
  7. 2025Mitigating Hallucinations in Large Vision-Language Models with Internal Fact-based Contrastive Decoding
  8. 2025Mixture of Decoding: An Attention-Inspired Adaptive Decoding Strategy to Mitigate Hallucinations in Large Vision-Language Models