mlsanityData to 14 Oct 2026

Attention Reallocation: Towards Zero-cost and Controllable Hallucination Mitigation of MLLMs

arXiv 2503.08342 ↗First posted 11 Mar 2025cs.CVcited by 23

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Multi-Modal Large Language Models (MLLMs) stand out in various tasks but still struggle with hallucinations. While recent training-free mitigation methods mostly introduce additional inference overhead via retrospection strategy and contrastive decoding, we propose attention reallocation (AttnReal) to mitigate hallucinations with nearly zero extra cost. Our approach is motivated by the key observations that, MLLM's unreasonable attention distribution causes features to be dominated by historical output tokens, which further contributes to hallucinated responses because of the distribution gap between different token types. Based on the observations, AttnReal recycles excessive attention from output tokens and reallocates it to visual tokens, which reduces MLLM's reliance on language priors and ensures the decoding process depends more on the visual inputs. More interestingly, we find that, by controlling the intensity of AttnReal, we can achieve a wide-range trade-off between the response faithfulness and overall performance. Comprehensive results from different benchmarks validate the effectiveness of AttnReal across six open-source MLLMs and three decoding strategies.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 50 of its 63 references that are in the map. First posted on arXiv 11 Mar 2025.

Builds on

50 of 63 references.

Closest ancestors · 20

Same direction, newest first

  1. 2025Mitigating Hallucinations in Large Vision-Language Models by Adaptively Constraining Information Flow28 Feb
  2. 2024Mitigating Object Hallucination via Concentric Causal Attention21 Oct
  3. 2024Mitigating Hallucination in Visual-Language Models via Re-Balancing Contrastive Decoding10 Sep
  4. 2024Alleviating Hallucination in Large Vision-Language Models with Active Retrieval Augmentation1 Aug
  5. 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMs31 Jul
  6. 2024Mitigating Object Hallucination in MLLMs via Data-augmented Phrase-level Alignment28 May
  7. 2024Don't Miss the Forest for the Trees: Attentional Vision Calibration for Large Vision Language Models28 May
  8. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding1 Mar
  9. 2023Hallucination Augmented Contrastive Learning for Multimodal Large Language Model12 Dec
  10. 2023Mitigating Fine-Grained Hallucination by Fine-Tuning Large Vision-Language Models with Caption Rewrites4 Dec
  11. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback1 Dec
  12. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
8 more
  1. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  2. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation13 Nov
  3. 2023Volcano: Mitigating Multimodal Hallucination through Self-Feedback Guided Revision13 Nov
  4. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Models24 Oct
  5. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models23 Oct
  6. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Models1 Oct
  7. 2023Detecting and Preventing Hallucinations in Large Vision Language Models11 Aug
  8. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun

Other references · 30

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  7. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  8. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  10. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  11. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  12. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
18 more
  1. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  2. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  3. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  4. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  5. 2018Object Hallucination in Image Captioningcited by 669
  6. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 547
  7. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 376
  8. 2023LMDrive: Closed-Loop End-to-End Driving with Large Language Modelscited by 305
  9. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 290
  10. 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Drivingcited by 232
  11. 2023LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmarkcited by 169
  12. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 163
  13. 2023ChartLlama: A Multimodal LLM for Chart Understanding and Generationcited by 163
  14. 2023ChatCAD: Interactive Computer-Aided Diagnosis on Medical Image using Large Language Modelscited by 116
  15. 2024MOOSE-Chem: Large Language Models for Rediscovering Unseen Chemistry Scientific Hypothesescited by 72
  16. 2023Vista-LLaMA: Reducing Hallucination in Video Language Models via Equal Distance to Visual Tokenscited by 71
  17. 2024MADTP: Multimodal Alignment-Guided Dynamic Token Pruning for Accelerating Vision-Language Transformercited by 54
  18. 2024ClipSAM: CLIP and SAM Collaboration for Zero-Shot Anomaly Segmentationcited by 49

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 16.

23 papers.

Sort:

By area:

  1. 2025Look-Back: Implicit Visual Re-focusing in MLLM Reasoningcited by 55
  2. 2025More Thought, Less Accuracy? On the Dual Nature of Reasoning in Vision-Language Modelscited by 27
  3. 2025FAVOR-Bench: A Comprehensive Benchmark for Fine-Grained Video Motion Understandingcited by 24
  4. 2025DeepSketcher: Internalizing Visual Manipulation for Multimodal Reasoningcited by 14
  5. 2025MaskCD: Mitigating LVLM Hallucinations by Image Head Masked Contrastive Decodingcited by 6
  6. 2026KVSmooth: Mitigating Hallucination in Multi-modal Large Language Models through Key-Value Smoothingcited by 5
  7. 2026OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyondcited by 5
  8. 2026Correcting Visual Blur Induced by Attention Distraction to Reduce Hallucinations: Algorithm and Theorycited by 3
  9. 2025MAP: Mitigating Hallucinations in Large Vision-Language Models with Map-Level Attention Processingcited by 1
  10. 2025Enhancing Medical Large Vision-Language Models via Alignment Distillationcited by 1
  11. 2026GEASS: Gated Evidence-Adaptive Selective Caption Trust for Vision-Language Modelscited by 1
  12. 2026Sink vs. diagonal patterns as mechanisms for attention switch and oversmoothing preventioncited by 1
11 more
  1. 2026CounterCount: A Diagnostic Framework for Counting Bias in Vision Language Modelscited by 1
  2. 2026RAVE: Re-Allocating Visual Attention in Large Multimodal Modelscited by 1
  3. 2026LASER: A Corrective Lens for LVLMs via Visual Attention Preservation and Sink Suppressioncited by 1
  4. 2026Attention to details, logits to truth: visual-aware attention and logits enhancement to mitigate hallucinations in LVLMscited by 0
  5. 2026Schr"oMind: Mitigating Hallucinations in Multimodal Large Language Models via Solving the Schr"odinger Bridge Problemcited by 0
  6. 2026Scalpel: Fine-Grained Alignment of Attention Activation Manifolds via Mixture Gaussian Bridges to Mitigate Multimodal Hallucinationcited by 0
  7. 2026Rethinking Visual Neglect: Steering via Context-Preference for MLLM Hallucination Mitigationcited by 0
  8. 2026Synergistic Perception-Reasoning Governance: Grounding Medical MLLMs with Verifiable Anatomical Evidencecited by 0
  9. 2026Role-Break in Attention Heads: Understanding and Detecting Hallucinations in VLMscited by 0
  10. 2026Beyond Tokens: A Survey on Decoding Methods for Large Language and Vision-Language Modelscited by 0
  11. 2026Look Before You Judge: Training-Free Region Mining for Grounded and Explainable Deepfake Detectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MINT: Mitigating Hallucinations in Large Vision-Language Models via Token Reduction
  2. 2025VEGAS: Mitigating Hallucinations in Large Vision-Language Models via Vision-Encoder Attention Guided Adaptive Steering
  3. 2025Mitigating Hallucinations in Vision-Language Models through Image-Guided Head Suppression
  4. 2024MLLM can see? Dynamic Correction Decoding for Hallucination Mitigation
  5. 2025Not All Tokens and Heads Are Equally Important: Dual-Level Attention Intervention for Hallucination Mitigation
  6. 2025TARAC: Mitigating Hallucination in LVLMs via Temporal Attention Real-time Accumulative Connection
  7. 2026Self-Correction Inside the Model: Leveraging Layer Attention to Mitigate Hallucinations in Large Vision Language Models
  8. 2024Mitigating Hallucinations in Large Vision-Language Models via Summary-Guided Decoding