mlsanityData to 7 Oct 2026

DAMRO: Dive into the Attention Mechanism of LVLM to Reduce Object Hallucination

arXiv 2410.04514 ↗First posted 6 Oct 2024cs.CL cs.CVcited by 47

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Despite the great success of Large Vision-Language Models (LVLMs), they inevitably suffer from hallucination. As we know, both the visual encoder and the Large Language Model (LLM) decoder in LVLMs are Transformer-based, allowing the model to extract visual information and generate text outputs via attention mechanisms. We find that the attention distribution of LLM decoder on image tokens is highly consistent with the visual encoder and both distributions tend to focus on particular background tokens rather than the referred objects in the image. We attribute to the unexpected attention distribution to an inherent flaw in the visual encoder itself, which misguides LLMs to over emphasize the redundant information and generate object hallucination. To address the issue, we propose DAMRO, a novel training-free strategy that Dive into Attention Mechanism of LVLM to Reduce Object Hallucination. Specifically, our approach employs classification token (CLS) of ViT to filter out high-attention outlier tokens scattered in the background and then eliminate their influence during decoding stage. We evaluate our method on LVLMs including LLaVA-1.5, LLaVA-NeXT and InstructBLIP, using various benchmarks such as POPE, CHAIR, MME and GPT-4V Aided Evaluation. The results demonstrate that our approach significantly reduces the impact of these outlier tokens, thus effectively alleviating the hallucination of LVLMs. The code is released at https://github.com/coder-gx/DAMRO.

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 17 of its 29 references that are in the map. First posted on arXiv 6 Oct 2024.

Builds on

17 of 29 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Detecting and Mitigating Hallucination in Large Vision Language Models via Fine-Grained AI Feedback22 Apr
  2. 2024Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding27 Mar
  3. 2024Multi-Modal Hallucination Control by Visual Information Grounding20 Mar
  4. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding1 Mar
  5. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  6. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov

Other references · 11

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2020End-to-End Object Detection with Transformerscited by 7,384
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  6. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  7. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  8. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  9. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 598
  10. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  11. 2023Creating Trustworthy LLMs: Dealing with Hallucinations in Healthcare AIcited by 33

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 25; 2026: 20.

47 papers.

Sort:

By area:

  1. 2025Mitigating Hallucinations in Large Vision-Language Models via DPO: On-Policy Data Hold the Keycited by 64
  2. 2025Large Vision-Language Model Alignment and Misalignment: A Survey Through the Lens of Explainabilitycited by 28
  3. 2025More Thought, Less Accuracy? On the Dual Nature of Reasoning in Vision-Language Modelscited by 26
  4. 2025Visual Attention Never Fades: Selective Progressive Attention ReCalibration for Detailed Image Captioning in Multimodal Large Language Modelscited by 21
  5. 2025Mitigating Hallucinations in Vision-Language Models through Image-Guided Head Suppressioncited by 21
  6. 2024VidHal: Benchmarking Temporal Hallucinations in Vision LLMscited by 19
  7. 2024Cracking the Code of Hallucination in LVLMs with Vision-aware Head Divergencecited by 19
  8. 2025Hallucinatory Image Tokens: A Training-free EAZY Approach on Detecting and Mitigating Object Hallucinations in LVLMscited by 14
  9. 2025Visual Multi-Agent System: Mitigating Hallucination Snowballing via Visual Flowcited by 13
  10. 2025TruthPrInt: Mitigating Large Vision-Language Models Object Hallucination Via Latent Truthful-Guided Pre-Interventioncited by 11
  11. 2025MCA-LLaVA: Manhattan Causal Attention for Reducing Hallucination in Large Vision-Language Modelscited by 11
  12. 2025GLSim: Detecting Object Hallucinations in LVLMs via Global-Local Similaritycited by 11
35 more
  1. 2025MedHEval: Benchmarking Hallucinations and Mitigation Strategies in Medical Large Vision-Language Modelscited by 10
  2. 2025Attention Hijackers: Detect and Disentangle Attention Hijacking in LVLMs for Hallucination Mitigationcited by 9
  3. 2026Hallucination Begins Where Saliency Dropscited by 8
  4. 2025Focus on What Matters: Enhancing Medical Vision-Language Models with Automatic Attention Alignment Tuningcited by 3
  5. 2025Revisit What You See: Revealing Visual Semantics in Vision Tokens to Guide LVLM Decodingcited by 3
  6. 2025CAI: Caption-Sensitive Attention Intervention for Mitigating Object Hallucination in Large Vision-Language Modelscited by 3
  7. 2025Cure or Poison? Embedding Instructions Visually Alters Hallucination in Vision-Language Modelscited by 3
  8. 2025Tracing and Mitigating Hallucinations in Multimodal LLMs via Dynamic Attention Localizationcited by 3
  9. 2025Zero-Shot Fine-Grained Image Classification Using Large Vision-Language Modelscited by 3
  10. 2025Reallocating Attention Across Layers to Reduce Multimodal Hallucinationcited by 3
  11. 2026Same Attention, Different Truths: Put Logit-Lens over Visual Attention to Detect and Mitigate LVLM Object Hallucinationcited by 3
  12. 2025VEGAS: Mitigating Hallucinations in Large Vision-Language Models via Vision-Encoder Attention Guided Adaptive Steeringcited by 2
  13. 2025MAP: Mitigating Hallucinations in Large Vision-Language Models with Map-Level Attention Processingcited by 1
  14. 2025Diving into Mitigating Hallucinations from a Vision Perspective for Large Vision-Language Modelscited by 1
  15. 2025PAS : Prelim Attention Score for Detecting Object Hallucinations in Large Vision--Language Modelscited by 1
  16. 2025Enhancing Medical Large Vision-Language Models via Alignment Distillationcited by 1
  17. 2026MACD: Model-Aware Contrastive Decoding via Counterfactual Datacited by 1
  18. 2026SAKED: Mitigating Hallucination in Large Vision-Language Models via Stability-Aware Knowledge Enhanced Decodingcited by 1
  19. 2026Instruction Lens Score: Your Instruction Contributes a Powerful Object Hallucination Detector for Multimodal Large Language Modelscited by 1
  20. 2026Propose and Attend: Training-free MLLM Grounding Confidence via Multi-Token Localized Attentioncited by 1
  21. 2025Instruction-Aligned Visual Attention for Mitigating Hallucinations in Large Vision-Language Modelscited by 0
  22. 2026System-Mediated Attention Imbalances Make Vision-Language Models Say Yescited by 0
  23. 2026Segmentation-Based Attention Entropy: Detecting and Mitigating Object Hallucinations in Large Vision-Language Modelscited by 0
  24. 2026Revealing Multi-View Hallucination in Large Vision-Language Modelscited by 0
  25. 2026CAST: Mitigating Object Hallucination in Large Vision-Language Models via Caption-Guided Visual Attention Steeringcited by 0
  26. 2026Uncertainty-Aware Exploratory Direct Preference Optimization for Multimodal Large Language Modelscited by 0
  27. 2026Through the Lens of Character: Resolving Modality-Role Interference in Multimodal Role-Playing Agentcited by 0
  28. 2026Mitigating Hallucination in Vision-Language Models through Barrier-Regulated Adaptive Closed-form Steeringcited by 0
  29. 2026MultiToP: Learning to Patch Visual Tokens to Mitigate Hallucinations in Video Large Multimodal Modelscited by 0
  30. 2026Synergistic Perception-Reasoning Governance: Grounding Medical MLLMs with Verifiable Anatomical Evidencecited by 0
  31. 2026LASER: A Corrective Lens for LVLMs via Visual Attention Preservation and Sink Suppressioncited by 0
  32. 2026When Visual Signals Mislead: A Mechanistic Study of Attribute Hallucination in Vision-Language Modelscited by 0
  33. 2026ReWEIGH the Evidence: Calibrating Token-Level Ordinal Visual Evidence to Mitigate Hallucinations in Large Vision-Language Modelscited by 0
  34. 2026VisER: Visual Evidence and Reliance for Object Hallucination Detection in LVLMscited by 0
  35. 2026Beyond Saying Less: Fine-Grained Alignment for Informative and Faithful Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Models
  2. 2025MINT: Mitigating Hallucinations in Large Vision-Language Models via Token Reduction
  3. 2026Reducing Object Hallucination in LVLMs via Emphasizing Image-negative Tokens
  4. 2026NoLan: Mitigating Object Hallucinations in Large Vision-Language Models via Dynamic Suppression of Language Priors
  5. 2023Evaluating Object Hallucination in Large Vision-Language Models
  6. 2025Modality Bias in LVLMs: Analyzing and Mitigating Object Hallucination via Attention Lens
  7. 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMs
  8. 2025PAINT: Paying Attention to INformed Tokens to Mitigate Hallucination in Large Vision-Language Model