mlsanityData to 14 Oct 2026

Mitigating Hallucinations in Large Vision-Language Models via Summary-Guided Decoding

arXiv 2410.13321 ↗First posted 17 Oct 2024cs.AI cs.CL cs.CVcited by 22

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Large Vision-Language Models (LVLMs) demonstrate impressive capabilities in generating detailed and coherent responses from visual inputs. However, they are prone to generate hallucinations due to an over-reliance on language priors. To address this issue, we investigate the language priors in LVLMs and make two key observations: (1) Even when predicting the tokens associated with image-related part-of-speech (POS), models increasingly rely on linguistic priors as the token sequences grow, thereby amplifying hallucinations. (2) Methods that directly calibrate LVLM's output distribution to mitigate language priors can lead to a degradation in text quality or even exacerbate hallucinations. Based on these findings, we propose a novel method, Summary-Guided Decoding (SumGD). This method naturally encourages the model to focus more on image information by reducing the text context through summaries, while controlling only the image-related POS tokens to maintain text quality. Through experiments, we demonstrate that SumGD achieves state-of-the-art performance on object hallucination benchmarks. Furthermore, in terms of the trade-off between precision and recall, SumGD achieves Pareto optimality among the existing methods. Lastly, we observe that although existing methods struggle to balance the reduction of object hallucinations with maintaining text quality, SumGD demonstrates robustness in handling this challenge.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 20 of its 31 references that are in the map. First posted on arXiv 17 Oct 2024.

Builds on

20 of 31 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Mitigating Hallucinations in Large Vision-Language Models (LVLMs) via Language-Contrastive Decoding (LCD)6 Aug
  2. 2024Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding27 Mar
  3. 2024Multi-Modal Hallucination Control by Visual Information Grounding20 Mar
  4. 2024IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decoding28 Feb
  5. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  6. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  7. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization28 Nov
  8. 2023Volcano: Mitigating Multimodal Hallucination through Self-Feedback Guided Revision13 Nov
  9. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun

Other references · 11

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  2. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  3. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  4. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  5. 2018Object Hallucination in Image Captioningcited by 669
  6. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 609
  7. 2024Contrastive Region Guidance: Improving Grounding in Vision-Language Models without Trainingcited by 64
  8. 2024Examining Gender and Racial Bias in Large Vision-Language Models Using a Novel Dataset of Parallel Imagescited by 55
  9. 2022Critic-Guided Decoding for Controlled Text Generationcited by 45
  10. 2022Detoxifying Text with MaRCo: Controllable Revision with Experts and Anti-Expertscited by 36
  11. 2024Can LLMs Recognize Toxicity? A Structured Investigation Framework and Toxicity Metriccited by 23

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 9; 2026: 12.

22 papers.

Sort:

By area:

  1. 2025Evaluating and Steering Modality Preferences in Multimodal Large Language Modelcited by 33
  2. 2024Are LLM-Judges Robust to Expressions of Uncertainty? Investigating the effect of Epistemic Markers on LLM-based Evaluationcited by 29
  3. 2025Large Vision-Language Model Alignment and Misalignment: A Survey Through the Lens of Explainabilitycited by 28
  4. 2026Hallucination Begins Where Saliency Dropscited by 9
  5. 2026Vision-Language Introspection: Mitigating Overconfident Hallucinations in MLLMs via Interpretable Bi-Causal Steeringcited by 8
  6. 2025Hallucinate at the Last in Long Response Generation: A Case Study on Long Document Summarizationcited by 6
  7. 2025Capturing Gaze Shifts for Guidance: Cross-Modal Fusion Enhancement for VLM Hallucination Mitigationcited by 5
  8. 2026Residual Decoding: Mitigating Hallucinations in Large Vision-Language Models via History-Aware Residual Guidancecited by 5
  9. 2025CCD: Mitigating Hallucinations in Radiology MLLMs via Clinical Contrastive Decodingcited by 4
  10. 2025Can Large Multimodal Models Actively Recognize Faulty Inputs? A Systematic Evaluation Framework of Their Input Scrutiny Abilitycited by 3
  11. 2025Grounding or Guessing? Visual Signals for Detecting Hallucinations in Sign Language Translationcited by 3
  12. 2026Decoding by Perturbation: Mitigating MLLM Hallucinations via Dynamic Textual Perturbationcited by 3
10 more
  1. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Surveycited by 2
  2. 2026Improving Medical Visual Reinforcement Fine-Tuning via Perception and Reasoning Augmentationcited by 1
  3. 2026First Logit Boosting: Visual Grounding Method to Mitigate Object Hallucination in Large Vision-Language Modelscited by 1
  4. 2026ReflectCAP: Detailed Image Captioning with Reflective Memorycited by 1
  5. 2025Rethinking Post-Unlearning Behavior of Large Vision-Language Modelscited by 0
  6. 2026See Fair, Speak Truth: Equitable Attention Improves Grounding and Reduces Hallucination in Vision-Language Alignmentcited by 0
  7. 2026Spectral Query-Key Product Weight Steering for Training-Free VLM Hallucination Mitigationcited by 0
  8. 2026LookBack: Where and How to Score LVLM Responses via Visual Reference Usagecited by 0
  9. 2026Beyond Tokens: A Survey on Decoding Methods for Large Language and Vision-Language Modelscited by 0
  10. 2026From Token Correction to Claim Admission: Training-Free Hallucination Mitigation for Large Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026NoLan: Mitigating Object Hallucinations in Large Vision-Language Models via Dynamic Suppression of Language Priors
  2. 2026FADE: Mitigating Hallucinations by Reducing Language-Prior Dominance in Large Vision-Language Models
  3. 2025Grounding Language with Vision: A Conditional Mutual Information Calibrated Decoding Strategy for Reducing Hallucinations in LVLMs
  4. 2024Seeing is Believing: Mitigating Hallucination in Large Vision-Language Models via CLIP-Guided Decoding
  5. 2025MINT: Mitigating Hallucinations in Large Vision-Language Models via Token Reduction
  6. 2025ONLY: One-Layer Intervention Sufficiently Mitigates Hallucinations in Large Vision-Language Models
  7. 2024Reducing Hallucinations in Vision-Language Models via Latent Space Steering
  8. 2025Mixture of Decoding: An Attention-Inspired Adaptive Decoding Strategy to Mitigate Hallucinations in Large Vision-Language Models