mlsanityData to 14 Oct 2026

MDSAM:Memory-Driven Sparse Attention Matrix for LVLMs Hallucination Mitigation

arXiv 2506.17664 ↗First posted 21 Jun 2025cs.CVcited by 1

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Hallucinations in large vision-language models (LVLMs) often stem from the model's sensitivity to image tokens during decoding, as evidenced by attention peaks observed when generating both real and hallucinated entities. To address this, we propose Memory-Driven Sparse Attention Matrix (MDSAM) , a novel training-free approach that dynamically captures and refines the attention allocated to image tokens at each layer. MDSAM memorizes attention patterns and activates updates through alignment during decoding, enhancing focus on relevant image tokens while effectively reducing hallucinations. We evaluate MDSAM on multiple benchmarks for tasks such as image captioning and visual question answering, demonstrating its ability to consistently reduce hallucinations and improve reliability. Compatible with various LVLM architectures, MDSAM highlights its adaptability and effectiveness in mitigating hallucinations without requiring additional training or external tools.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 24 of its 33 references that are in the map. First posted on arXiv 21 Jun 2025.

Builds on

24 of 33 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  2. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  3. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization28 Nov
  4. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun
  5. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 19

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  5. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  6. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  7. 2020Learning to summarize from human feedbackcited by 2,588
  8. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  9. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  10. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  11. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  12. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 853
7 more
  1. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 733
  2. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  3. 2018Object Hallucination in Image Captioningcited by 669
  4. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 572
  5. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 365
  6. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  7. 2023Caption Anything: Interactive Image Description with Diverse Multimodal Controlscited by 99

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Persistent Visual Memory: Sustaining Perception for Deep Generation in LVLMscited by 4

Similar, not cited

Close in wording, no citation either way

  1. 2026MHSA: A Lightweight Framework for Mitigating Hallucinations via Steered Attention in LVLMs
  2. 2025MINT: Mitigating Hallucinations in Large Vision-Language Models via Token Reduction
  3. 2026Attention to details, logits to truth: visual-aware attention and logits enhancement to mitigate hallucinations in LVLMs
  4. 2025VEGAS: Mitigating Hallucinations in Large Vision-Language Models via Vision-Encoder Attention Guided Adaptive Steering
  5. 2025Mitigating Hallucinations in Vision-Language Models through Image-Guided Head Suppression
  6. 2026See Only When Needed: Context-Aware Attention Intervention for Mitigating Hallucinations in LVLMs
  7. 2025Mixture of Decoding: An Attention-Inspired Adaptive Decoding Strategy to Mitigate Hallucinations in Large Vision-Language Models
  8. 2024Seeing Clearly by Layer Two: Enhancing Attention Heads to Alleviate Hallucination in LVLMs