# DAMRO: Dive into the Attention Mechanism of LVLM to Reduce Object Hallucination

arXiv: https://arxiv.org/abs/2410.04514 · HTML: https://mlsanity.com/paper/2410.04514/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 17 of its 29 references that are in the map.
- First posted on arXiv 6 Oct 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Hallucination mitigation in LVLMs](https://mlsanity.com/mllm/hallucination-mitigation-in-lvlms/)

## Builds on (most cited first, 17 of 17)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [EVA-CLIP: Improved Training Techniques for CLIP at Scale](https://mlsanity.com/paper/2303.15389/) (2023)
- [Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs](https://mlsanity.com/paper/2401.06209/) (2024)
- [Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding](https://mlsanity.com/paper/2311.16922/) (2023)
- [Contrastive Decoding: Open-ended Text Generation as Optimization](https://mlsanity.com/paper/2210.15097/) (2022)
- [mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration](https://mlsanity.com/paper/2311.04257/) (2023)
- [OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation](https://mlsanity.com/paper/2311.17911/) (2023)
- [Multi-Modal Hallucination Control by Visual Information Grounding](https://mlsanity.com/paper/2403.14003/) (2024)
- [Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding](https://mlsanity.com/paper/2403.18715/) (2024)
- [HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding](https://mlsanity.com/paper/2403.00425/) (2024)
- [Detecting and Mitigating Hallucination in Large Vision Language Models via Fine-Grained AI Feedback](https://mlsanity.com/paper/2404.14233/) (2024)
- [Creating Trustworthy LLMs: Dealing with Hallucinations in Healthcare AI](https://mlsanity.com/paper/2311.01463/) (2023)

## Built on by (most cited first, 30 of 48)

- [Mitigating Hallucinations in Large Vision-Language Models via DPO: On-Policy Data Hold the Key](https://mlsanity.com/paper/2501.09695/) (2025)
- [Large Vision-Language Model Alignment and Misalignment: A Survey Through the Lens of Explainability](https://mlsanity.com/paper/2501.01346/) (2025)
- [More Thought, Less Accuracy? On the Dual Nature of Reasoning in Vision-Language Models](https://mlsanity.com/paper/2509.25848/) (2025)
- [Visual Attention Never Fades: Selective Progressive Attention ReCalibration for Detailed Image Captioning in Multimodal Large Language Models](https://mlsanity.com/paper/2502.01419/) (2025)
- [Mitigating Hallucinations in Vision-Language Models through Image-Guided Head Suppression](https://mlsanity.com/paper/2505.16411/) (2025)
- [VidHal: Benchmarking Temporal Hallucinations in Vision LLMs](https://mlsanity.com/paper/2411.16771/) (2024)
- [Cracking the Code of Hallucination in LVLMs with Vision-aware Head Divergence](https://mlsanity.com/paper/2412.13949/) (2024)
- [Hallucinatory Image Tokens: A Training-free EAZY Approach on Detecting and Mitigating Object Hallucinations in LVLMs](https://mlsanity.com/paper/2503.07772/) (2025)
- [Visual Multi-Agent System: Mitigating Hallucination Snowballing via Visual Flow](https://mlsanity.com/paper/2509.21789/) (2025)
- [MCA-LLaVA: Manhattan Causal Attention for Reducing Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2507.09184/) (2025)
- [TruthPrInt: Mitigating Large Vision-Language Models Object Hallucination Via Latent Truthful-Guided Pre-Intervention](https://mlsanity.com/paper/2503.10602/) (2025)
- [GLSim: Detecting Object Hallucinations in LVLMs via Global-Local Similarity](https://mlsanity.com/paper/2508.19972/) (2025)
- [MedHEval: Benchmarking Hallucinations and Mitigation Strategies in Medical Large Vision-Language Models](https://mlsanity.com/paper/2503.02157/) (2025)
- [Attention Hijackers: Detect and Disentangle Attention Hijacking in LVLMs for Hallucination Mitigation](https://mlsanity.com/paper/2503.08216/) (2025)
- [Hallucination Begins Where Saliency Drops](https://mlsanity.com/paper/2601.20279/) (2026)
- [Revisit What You See: Revealing Visual Semantics in Vision Tokens to Guide LVLM Decoding](https://mlsanity.com/paper/2506.09522/) (2025)
- [Cure or Poison? Embedding Instructions Visually Alters Hallucination in Vision-Language Models](https://mlsanity.com/paper/2508.01678/) (2025)
- [Tracing and Mitigating Hallucinations in Multimodal LLMs via Dynamic Attention Localization](https://mlsanity.com/paper/2509.07864/) (2025)
- [Zero-Shot Fine-Grained Image Classification Using Large Vision-Language Models](https://mlsanity.com/paper/2510.03903/) (2025)
- [Reallocating Attention Across Layers to Reduce Multimodal Hallucination](https://mlsanity.com/paper/2510.10285/) (2025)
- [Same Attention, Different Truths: Put Logit-Lens over Visual Attention to Detect and Mitigate LVLM Object Hallucination](https://mlsanity.com/paper/2608.07302/) (2026)
- [Focus on What Matters: Enhancing Medical Vision-Language Models with Automatic Attention Alignment Tuning](https://mlsanity.com/paper/2505.18503/) (2025)
- [CAI: Caption-Sensitive Attention Intervention for Mitigating Object Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2506.23590/) (2025)
- [VEGAS: Mitigating Hallucinations in Large Vision-Language Models via Vision-Encoder Attention Guided Adaptive Steering](https://mlsanity.com/paper/2512.12089/) (2025)
- [MACD: Model-Aware Contrastive Decoding via Counterfactual Data](https://mlsanity.com/paper/2602.01740/) (2026)
- [Propose and Attend: Training-free MLLM Grounding Confidence via Multi-Token Localized Attention](https://mlsanity.com/paper/2607.05978/) (2026)
- [MAP: Mitigating Hallucinations in Large Vision-Language Models with Map-Level Attention Processing](https://mlsanity.com/paper/2508.01653/) (2025)
- [Diving into Mitigating Hallucinations from a Vision Perspective for Large Vision-Language Models](https://mlsanity.com/paper/2509.13836/) (2025)
- [PAS : Prelim Attention Score for Detecting Object Hallucinations in Large Vision--Language Models](https://mlsanity.com/paper/2511.11502/) (2025)
- [Enhancing Medical Large Vision-Language Models via Alignment Distillation](https://mlsanity.com/paper/2512.18554/) (2025)

## Abstract (arXiv)

Despite the great success of Large Vision-Language Models (LVLMs), they inevitably suffer from hallucination. As we know, both the visual encoder and the Large Language Model (LLM) decoder in LVLMs are Transformer-based, allowing the model to extract visual information and generate text outputs via attention mechanisms. We find that the attention distribution of LLM decoder on image tokens is highly consistent with the visual encoder and both distributions tend to focus on particular background tokens rather than the referred objects in the image. We attribute to the unexpected attention distribution to an inherent flaw in the visual encoder itself, which misguides LLMs to over emphasize the redundant information and generate object hallucination. To address the issue, we propose DAMRO, a novel training-free strategy that Dive into Attention Mechanism of LVLM to Reduce Object Hallucination. Specifically, our approach employs classification token (CLS) of ViT to filter out high-attention outlier tokens scattered in the background and then eliminate their influence during decoding stage. We evaluate our method on LVLMs including LLaVA-1.5, LLaVA-NeXT and InstructBLIP, using various benchmarks such as POPE, CHAIR, MME and GPT-4V Aided Evaluation. The results demonstrate that our approach significantly reduces the impact of these outlier tokens, thus effectively alleviating the hallucination of LVLMs. The code is released at https://github.com/coder-gx/DAMRO.