mlsanityData to 7 Oct 2026

Visual Description Grounding Reduces Hallucinations and Boosts Reasoning in LVLMs

arXiv 2405.15683 ↗First posted 24 May 2024cs.CV cs.AI cs.CLcited by 33

On the map: LLM reasoning → Hallucination detection and mitigation

also in 1 more
Abstract (arXiv)

Large Vision-Language Models (LVLMs) often produce responses that misalign with factual information, a phenomenon known as hallucinations. While hallucinations are well-studied, the exact causes behind them remain underexplored. In this paper, we first investigate the root causes of hallucinations in LVLMs. Our findings reveal that existing mitigation techniques primarily reduce hallucinations for visual recognition prompts-those that require simple descriptions of visual elements-but fail for cognitive prompts that demand deliberate reasoning. We identify the core issue as a lack of true visual perception in LVLMs: although they can accurately recognize visual elements, they struggle to fully interpret these elements in the context of the input prompt and effectively link this recognition to their internal knowledge, which is critical for reasoning. To address this gap, we introduce Visual Description Grounded Decoding (VDGD), a simple, robust, and training-free method designed to enhance visual perception and improve reasoning capabilities in LVLMs. VDGD works by first generating a detailed description of the image and appending it as a prefix to the instruction. During response generation, tokens are sampled based on their KL divergence to the description, favoring candidates with lower divergence. Experimental results on multiple visual reasoning benchmarks and LVLMs demonstrate that VDGD consistently outperforms existing baselines 2% - 33%. Finally, we introduce VaLLu, a benchmark designed for comprehensive evaluation of the cognitive capabilities of LVLMs.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Placed in Hallucination detection and mitigation, one of 17 directions in LLM reasoning. Builds on 24 of its 55 references that are in the map. First posted on arXiv 24 May 2024.

Builds on

24 of 55 references.

References · 24

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  3. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  4. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  5. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  6. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  7. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  8. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  9. 2023A Survey on Multimodal Large Language Modelscited by 898
  10. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  11. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
  12. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 572
12 more
  1. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 529
  2. 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Datasetcited by 516
  3. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 468
  4. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 380
  5. 2024A Survey on Hallucination in Large Vision-Language Modelscited by 352
  6. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Modelscited by 314
  7. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  8. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluationcited by 293
  9. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Modelscited by 223
  10. 2023MMC: Advancing Multimodal Chart Understanding with Large-scale Instruction Tuningcited by 149
  11. 2023Open-domain Visual Entity Recognition: Towards Recognizing Millions of Wikipedia Entitiescited by 112
  12. 2023Look-back Decoding for Open-Ended Text Generationcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 10; 2026: 19.

33 papers.

Sort:

By area:

  1. 2024Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Modelscited by 90
  2. 2025Grounded Reinforcement Learning for Visual Reasoningcited by 80
  3. 2025A Survey on Large Language Models for Mathematical Reasoningcited by 52
  4. 2025Large Vision-Language Model Alignment and Misalignment: A Survey Through the Lens of Explainabilitycited by 28
  5. 2025Mellow: a small audio language model for reasoningcited by 27
  6. 2026Visual-Advantage On-Policy Distillation for Vision-Language Modelscited by 22
  7. 2024Cracking the Code of Hallucination in LVLMs with Vision-aware Head Divergencecited by 19
  8. 2024Losing Visual Needles in Image Haystacks: Vision Language Models are Easily Distracted in Short and Long Contextscited by 14
  9. 2026HALP: Detecting Hallucinations in Vision-Language Models without Generating a Single Tokencited by 13
  10. 2025Faithful-First Reasoning, Planning, and Acting for Multimodal LLMscited by 7
  11. 2024Hallucination Elimination and Semantic Enhancement Framework for Vision-Language Models in Traffic Scenarioscited by 4
  12. 2025CCD: Mitigating Hallucinations in Radiology MLLMs via Clinical Contrastive Decodingcited by 4
21 more
  1. 2025Grounding or Guessing? Visual Signals for Detecting Hallucinations in Sign Language Translationcited by 3
  2. 2025Towards Mitigating Hallucinations in Large Vision-Language Models by Refining Textual Embeddingscited by 3
  3. 2026KVSmooth: Mitigating Hallucination in Multi-modal Large Language Models through Key-Value Smoothingcited by 3
  4. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Surveycited by 2
  5. 2026Ground What You See: Hallucination-Resistant MLLMs via Caption Feedback, Diversity-Aware Sampling, and Conflict Regularizationcited by 2
  6. 2026Looking Back and Forth: Cross-Image Attention Calibration and Attentive Preference Learning for Multi-Image Hallucination Mitigationcited by 2
  7. 2026Med-OPD: Improving Medical Vision-Language Models via Evidence-Aware On-Policy Distillationcited by 2
  8. 2026FaithSCAN: Model-Driven Single-Pass Hallucination Detection for Faithful Visual Question Answeringcited by 1
  9. 2026See It, Say It, Sorted: An Iterative Training-Free Framework for Visually-Grounded Multimodal Reasoning in LVLMscited by 1
  10. 2026To See or To Please: Uncovering Visual Sycophancy and Split Beliefs in VLMscited by 1
  11. 2026Risk-aware Selective Prompting for Hallucination Mitigation in Large Vision-Language Modelscited by 1
  12. 2025Rationale-Enhanced Decoding for Multi-modal Chain-of-Thoughtcited by 0
  13. 2026System-Mediated Attention Imbalances Make Vision-Language Models Say Yescited by 0
  14. 2026ClueTracer: Question-to-Vision Clue Tracing for Training-Free Hallucination Suppression in Multimodal Reasoningcited by 0
  15. 2026Revisiting Greedy Decoding for Visual Question Answering: A Calibration Perspectivecited by 0
  16. 2026GEASS: Gated Evidence-Adaptive Selective Caption Trust for Vision-Language Modelscited by 0
  17. 2026CARE: Competence-Aware Reward Shaping for Adaptive Reasoning Length in Video-MLLMscited by 0
  18. 2026Diagnosing and Mitigating Perception-Decision Misalignment in Omni-LLMs via Modality Subspace Activationcited by 0
  19. 2026Counterfactual Anatomy-guided Spatial-Temporal Decoding for Annotation-Free Hallucination Mitigation in Medical VLMscited by 0
  20. 2026When Words Speak Louder than Images: Towards Understanding Language Bias in Vision-Language Modelscited by 0
  21. 2026From Routing Signals to Selective Review: Visual regrounding in MoE VLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Combating Multimodal LLM Hallucination via Bottom-Up Holistic Reasoning
  2. 2026When Prompts Override Vision: Prompt-Induced Hallucinations in LVLMs
  3. 2024Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding
  4. 2024Reducing Hallucinations in Vision-Language Models via Latent Space Steering
  5. 2026Reducing Object Hallucination in LVLMs via Emphasizing Image-negative Tokens
  6. 2026See Only When Needed: Context-Aware Attention Intervention for Mitigating Hallucinations in LVLMs
  7. 2025When Images Speak Louder: Mitigating Language Bias-induced Hallucinations in VLMs through Cross-Modal Guidance
  8. 2024Does Object Grounding Really Reduce Hallucination of Large Vision-Language Models?