mlsanityData to 14 Oct 2026

MCA-LLaVA: Manhattan Causal Attention for Reducing Hallucination in Large Vision-Language Models

arXiv 2507.09184 ↗First posted 12 Jul 2025cs.CVcited by 12

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Hallucinations pose a significant challenge in Large Vision Language Models (LVLMs), with misalignment between multimodal features identified as a key contributing factor. This paper reveals the negative impact of the long-term decay in Rotary Position Encoding (RoPE), used for positional modeling in LVLMs, on multimodal alignment. Concretely, under long-term decay, instruction tokens exhibit uneven perception of image tokens located at different positions within the two-dimensional space: prioritizing image tokens from the bottom-right region since in the one-dimensional sequence, these tokens are positionally closer to the instruction tokens. This biased perception leads to insufficient image-instruction interaction and suboptimal multimodal alignment. We refer to this phenomenon as image alignment bias. To enhance instruction's perception of image tokens at different spatial locations, we propose MCA-LLaVA, based on Manhattan distance, which extends the long-term decay to a two-dimensional, multi-directional spatial decay. MCA-LLaVA integrates the one-dimensional sequence order and two-dimensional spatial position of image tokens for positional modeling, mitigating hallucinations by alleviating image alignment bias. Experimental results of MCA-LLaVA across various hallucination and general benchmarks demonstrate its effectiveness and generality. The code can be accessed in https://github.com/ErikZ719/MCA-LLaVA.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 47 of its 70 references that are in the map. First posted on arXiv 12 Jul 2025.

Builds on

47 of 70 references.

Closest ancestors · 20

Same direction, newest first

  1. 2025Seeing Far and Clearly: Mitigating Hallucinations in MLLMs with Attention Causal Decoding22 May
  2. 2025PerturboLLaVA: Reducing Multimodal Hallucinations with Perturbative Visual Training9 Mar
  3. 2025The Hidden Life of Tokens: Reducing Hallucination of Large Vision-Language Models via Visual Information Steering5 Feb
  4. 2024VaLiD: Mitigating the Hallucination of Large Vision Language Models by Visual Layer Fusion Contrastive Decoding24 Nov
  5. 2024V-DPO: Mitigating Hallucination in Large Vision Language Models via Vision-Guided Direct Preference Optimization5 Nov
  6. 2024The Curse of Multi-Modalities: Evaluating Hallucinations of Large Multimodal Models across Language, Visual, and Audio16 Oct
  7. 2024DAMRO: Dive into the Attention Mechanism of LVLM to Reduce Object Hallucination6 Oct
  8. 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMs31 Jul
  9. 2024DOPRA: Decoding Over-accumulation Penalization and Re-allocation in Specific Weighting Layer21 Jul
  10. 2024Mitigating Object Hallucination in MLLMs via Data-augmented Phrase-level Alignment28 May
  11. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding1 Mar
  12. 2024IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decoding28 Feb
8 more
  1. 2023Mitigating Fine-Grained Hallucination by Fine-Tuning Large Vision-Language Models with Caption Rewrites4 Dec
  2. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback1 Dec
  3. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  4. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  5. 2023HalluciDoctor: Mitigating Hallucinatory Toxicity in Visual Instruction Data22 Nov
  6. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Models1 Oct
  7. 2023Detecting and Preventing Hallucinations in Large Vision Language Models11 Aug
  8. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun

Other references · 27

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  5. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  6. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  7. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  8. 2019Towards VQA Models That Can Readcited by 1,718
  9. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  10. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  11. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
  12. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
15 more
  1. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  2. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  3. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  4. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  5. 2018Object Hallucination in Image Captioningcited by 669
  6. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 642
  7. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 547
  8. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 376
  9. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 302
  10. 2023Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learningcited by 236
  11. 2024Strengthening Multimodal Large Language Model with Bootstrapped Preference Optimizationcited by 75
  12. 2020Improve Transformer Models with Better Relative Position Embeddingscited by 73
  13. 2024CREAM: Consistency Regularized Self-Rewarding Language Modelscited by 35
  14. 2025MMRC: A Large-Scale Benchmark for Understanding Multimodal Large Language Model in Real-World Conversationcited by 29
  15. 2024Fine-Grained Verifiers: Preference Modeling as Next-token Prediction in Vision-Language Alignmentcited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 8.

12 papers.

  1. 2025SHIELD: Suppressing Hallucinations In LVLM Encoders via Bias and Vulnerability Defensecited by 7
  2. 2026Aligning What Vision-Language Models See and Perceive with Adaptive Information Flowcited by 3
  3. 2026Reroute, Don't Remove: Recoverable Visual Token Routing for Vision-Language Modelscited by 3
  4. 2026C^2ROPE: Causal Continuous Rotary Positional Encoding for 3D Large Multimodal-Models Reasoningcited by 2
  5. 2026SoPE: Spherical Coordinate-Based Positional Embedding for Enhancing Spatial Perception of 3D LVLMscited by 2
  6. 2026Inference Time Optimization with Confidence Dynamicscited by 2
  7. 2026Mitigating Object Hallucinations in LVLMs via Attention Imbalance Rectificationcited by 1
  8. 2025SFormer: SNR-guided Transformer for Underwater Image Enhancement from the Frequency Domaincited by 0
  9. 2025A Comprehensive Analysis of Churn Prediction in Telecommunications Using Machine Learningcited by 0
  10. 2025Forestpest-YOLO: A High-Performance Detection Framework for Small Forestry Pestscited by 0
  11. 2026DAPE: Dynamic Non-uniform Alignment and Progressive Detail Enhancement Techniques for Improving the Performance of Efficient Visual Language Modelscited by 0
  12. 2026Look Where You Say You're Looking: Self-Grounded Attention for Visual Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Mitigating Object Hallucination via Concentric Causal Attention
  2. 2025Mitigating Object Hallucinations in Large Vision-Language Models via Attention Calibration
  3. 2024Reducing Hallucinations in Vision-Language Models via Latent Space Steering
  4. 2024Understanding Multimodal Hallucination with Parameter-Free Representation Alignment
  5. 2024From Pixels to Tokens: Revisiting Object Hallucinations in Large Vision-Language Models
  6. 2024Cracking the Code of Hallucination in LVLMs with Vision-aware Head Divergence
  7. 2025MINT: Mitigating Hallucinations in Large Vision-Language Models via Token Reduction
  8. 2025Mitigating Hallucination in Large Vision-Language Models via Adaptive Attention Calibration