mlsanityData to 14 Oct 2026

MAP: Mitigating Hallucinations in Large Vision-Language Models with Map-Level Attention Processing

arXiv 2508.01653 ↗First posted 3 Aug 2025cs.CV cs.AIcited by 1

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Large Vision-Language Models (LVLMs) have achieved impressive performance in multimodal tasks, but they still suffer from hallucinations, i.e., generating content that is grammatically accurate but inconsistent with visual inputs. In this work, we introduce a novel map-level perspective to mitigate hallucinations in LVLMs, interpreting the hidden states of the model as a 2D semantic map. We observe that factual information is widely distributed across this map, extending beyond the localized inter- or intra-layer regions targeted by most existing methods (e.g., contrastive decoding and layer-wise consistency). Building on this insight, we propose Map-Level Attention Processing (MAP), a training-free decoding method that effectively leverages factual information through attention-based map-level operations to improve factual consistency. Specifically, we employ Layer-Wise Criss-Cross Attention to progressively refine token representations at each decoding layer by aggregating tokens from both inter- and intra-layer dimensions. Additionally, a Global-Local Logit Fusion mechanism combines logits obtained before and after global attention to further refine predictions and improve accuracy. Our method consistently improves the truthfulness and performance of LVLMs across benchmarks, such as POPE, MME, and MMHal-Bench, demonstrating the potential of the map-level decoding strategy.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 49 of its 55 references that are in the map. First posted on arXiv 3 Aug 2025.

Builds on

49 of 55 references.

Closest ancestors · 24

Same direction, newest first

  1. 2025HalluRNN: Mitigating Hallucinations via Recurrent Cross-Layer Reasoning in Large Vision-Language Models21 Jun
  2. 2025Mitigating Hallucinations in Vision-Language Models through Image-Guided Head Suppression22 May
  3. 2025Mitigating Hallucinations via Inter-Layer Consistency Aggregation in Large Vision-Language Models18 May
  4. 2025Attention Reallocation: Towards Zero-cost and Controllable Hallucination Mitigation of MLLMs11 Mar
  5. 2024Hallucination Elimination and Semantic Enhancement Framework for Vision-Language Models in Traffic Scenarios10 Dec
  6. 2024Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lens23 Nov
  7. 2024MLLM can see? Dynamic Correction Decoding for Hallucination Mitigation15 Oct
  8. 2024DAMRO: Dive into the Attention Mechanism of LVLM to Reduce Object Hallucination6 Oct
  9. 2024Alleviating Hallucination in Large Vision-Language Models with Active Retrieval Augmentation1 Aug
  10. 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMs31 Jul
  11. 2024MedVH: Towards Systematic Evaluation of Hallucination for Large Vision Language Models in the Medical Context3 Jul
  12. 2024ESREAL: Exploiting Semantic Reconstruction to Mitigate Hallucinations in Vision-Language Models24 Mar
12 more
  1. 2024IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decoding28 Feb
  2. 2023Hallucination Augmented Contrastive Learning for Multimodal Large Language Model12 Dec
  3. 2023Mitigating Fine-Grained Hallucination by Fine-Tuning Large Vision-Language Models with Caption Rewrites4 Dec
  4. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback1 Dec
  5. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  6. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  7. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization28 Nov
  8. 2023Mitigating Hallucination in Visual Language Models with Visual Supervision27 Nov
  9. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Models1 Oct
  10. 2023Evaluation and Analysis of Hallucination in Large Vision-Language Models29 Aug
  11. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun
  12. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 25

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  2. 2023Visual Instruction Tuningcited by 7,869
  3. 2025Qwen2.5-VL Technical Reportcited by 5,587
  4. 2024GPT-4o System Cardcited by 4,588
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  6. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  7. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  8. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  9. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  10. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  11. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  12. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 798
13 more
  1. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  2. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  3. 2018Object Hallucination in Image Captioningcited by 669
  4. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 547
  5. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 376
  6. 2023A Survey of Hallucination in Large Foundation Modelscited by 368
  7. 2025See What You Are Told: Visual Attention Sink in Large Multimodal Modelscited by 132
  8. 2024ViGoR: Improving Visual Grounding of Large Vision Language Models with Fine-Grained Reward Modelingcited by 34
  9. 2024FGAIF: Aligning Large Vision-Language Models with Fine-grained AI Feedbackcited by 30
  10. 2025Large Vision-Language Model Alignment and Misalignment: A Survey Through the Lens of Explainabilitycited by 28
  11. 2025Visual question answering: from early developments to recent advances -- a surveycited by 19
  12. 2024Mitigating Entity-Level Hallucination in Large Language Modelscited by 17
  13. 2025FilterRAG: Zero-Shot Informed Retrieval-Augmented Generation to Mitigate Hallucinations in VQAcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026FADE: Mitigating Hallucinations by Reducing Language-Prior Dominance in Large Vision-Language Modelscited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2025VEGAS: Mitigating Hallucinations in Large Vision-Language Models via Vision-Encoder Attention Guided Adaptive Steering
  2. 2025Mitigating Hallucination in Large Vision-Language Models through Aligning Attention Distribution to Information Flow
  3. 2024Reducing Hallucinations in Vision-Language Models via Latent Space Steering
  4. 2025Mixture of Decoding: An Attention-Inspired Adaptive Decoding Strategy to Mitigate Hallucinations in Large Vision-Language Models
  5. 2025MINT: Mitigating Hallucinations in Large Vision-Language Models via Token Reduction
  6. 2024Interpreting and Editing Vision-Language Representations to Mitigate Hallucinations
  7. 2024From Pixels to Tokens: Revisiting Object Hallucinations in Large Vision-Language Models
  8. 2025Mitigating Object Hallucinations in Large Vision-Language Models via Attention Calibration