mlsanityData to 14 Oct 2026

Do You Keep an Eye on What I Ask? Mitigating Multimodal Hallucination via Attention-Guided Ensemble Decoding

arXiv 2505.17529 ↗First posted 23 May 2025cs.CV cs.AIcited by 12

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Recent advancements in Large Vision-Language Models (LVLMs) have significantly expanded their utility in tasks like image captioning and visual question answering. However, they still struggle with object hallucination, where models generate descriptions that inaccurately reflect the visual content by including nonexistent objects or misrepresenting existing ones. While previous methods, such as data augmentation and training-free approaches, strive to tackle this issue, they still encounter scalability challenges and often depend on additional external modules. In this work, we propose Ensemble Decoding (ED), a novel strategy that splits the input image into sub-images and combines logit distributions by assigning weights through the attention map. Furthermore, we introduce ED adaptive plausibility constraint to calibrate logit distribution and FastED, a variant designed for speed-critical applications. Extensive experiments across hallucination benchmarks demonstrate that our proposed method achieves state-of-the-art performance, validating the effectiveness of our approach.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 46 of its 52 references that are in the map. First posted on arXiv 23 May 2025.

Builds on

46 of 52 references.

Closest ancestors · 16

Same direction, newest first

  1. 2024CLIP-DPO: Vision-Language Models as a Source of Preference for Fixing Hallucinations in LVLMs19 Aug
  2. 2024Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding27 Mar
  3. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding1 Mar
  4. 2024IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decoding28 Feb
  5. 2024Logical Closed Loop: Uncovering Object Hallucinations in Large Vision-Language Models18 Feb
  6. 2024A Survey on Hallucination in Large Vision-Language Models1 Feb
  7. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  8. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  9. 2023Mitigating Hallucination in Visual Language Models with Visual Supervision27 Nov
  10. 2023Volcano: Mitigating Multimodal Hallucination through Self-Feedback Guided Revision13 Nov
  11. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Models24 Oct
  12. 2023Negative Object Presence Evaluation (NOPE) to Measure Object Hallucination in Vision-Language Models9 Oct
4 more
  1. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Models1 Oct
  2. 2023Detecting and Preventing Hallucinations in Large Vision Language Models11 Aug
  3. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun
  4. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 30

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  8. 2023Mistral 7Bcited by 3,164
  9. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  10. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  11. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,331
  12. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
18 more
  1. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  2. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  3. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  4. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  5. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 698
  6. 2018Object Hallucination in Image Captioningcited by 669
  7. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 609
  8. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 376
  9. 2023HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Modelscited by 341
  10. 2023How Language Model Hallucinations Can Snowballcited by 311
  11. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 274
  12. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  13. 2023Honeybee: Locality-enhanced Projector for Multimodal LLMcited by 197
  14. 2020Detecting Hallucinated Content in Conditional Neural Sequence Generationcited by 152
  15. 2021Causal Attention for Vision-Language Taskscited by 111
  16. 2021Let there be a clock on the beach: Reducing Object Hallucination in Image Captioningcited by 76
  17. 2024Enhancing Visual Document Understanding with Contrastive Learning in Large Visual-Language Modelscited by 30
  18. 2023Safety of autonomous vehicles: A survey on Model-based vs. AI-based approachescited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 9.

12 papers.

  1. 2025SAVE: Sparse Autoencoder-Driven Visual Information Enhancement for Mitigating Object Hallucinationcited by 5
  2. 2026MAD: Modality-Adaptive Decoding for Mitigating Cross-Modal Hallucinations in Multimodal Large Language Modelscited by 5
  3. 2026KVSmooth: Mitigating Hallucination in Multi-modal Large Language Models through Key-Value Smoothingcited by 5
  4. 2026Mitigating Manifold Departure: Uncertainty-Aware Subspace Rectification for Trustworthy MLLM Decodingcited by 4
  5. 2025VEGAS: Mitigating Hallucinations in Large Vision-Language Models via Vision-Encoder Attention Guided Adaptive Steeringcited by 3
  6. 2026Decoding by Perturbation: Mitigating MLLM Hallucinations via Dynamic Textual Perturbationcited by 3
  7. 2026When Prompts Override Vision: Prompt-Induced Hallucinations in LVLMscited by 2
  8. 2026ADAPT: Attention Dynamics Alignment with Preference Tuning for Faithful MLLMscited by 2
  9. 2025Beyond Single Models: Mitigating Multimodal Hallucinations via Adaptive Token Ensemble Decodingcited by 1
  10. 2026Clearer Sight, Fewer Lies: Oriented Pickup Preference Optimization for Multimodal Hallucination Mitigationcited by 1
  11. 2026Visual Attention Drifts,but Anchors Hold:Mitigating Hallucination in Multimodal Large Language Models via Cross-Layer Visual Anchorscited by 0
  12. 2026Beyond Attention Imbalance: Mitigating Hallucinations via Spectral Surgerycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Mixture of Decoding: An Attention-Inspired Adaptive Decoding Strategy to Mitigate Hallucinations in Large Vision-Language Models
  2. 2024Seeing is Believing: Mitigating Hallucination in Large Vision-Language Models via CLIP-Guided Decoding
  3. 2025CAI: Caption-Sensitive Attention Intervention for Mitigating Object Hallucination in Large Vision-Language Models
  4. 2026Revealing Multi-View Hallucination in Large Vision-Language Models
  5. 2026Mask What Matters: Mitigating Object Hallucinations in Multimodal Large Language Models with Object-Aligned Visual Contrastive Decoding
  6. 2026NoLan: Mitigating Object Hallucinations in Large Vision-Language Models via Dynamic Suppression of Language Priors
  7. 2026Mitigating Object Hallucinations in Vision-Language Models through Region-Aware Attention Recalibration
  8. 2023HallE-Control: Controlling Object Hallucination in Large Multimodal Models