mlsanityData to 14 Oct 2026

Optimizing Vision-Language Consistency via Cross-Layer Regional Attention Alignment

arXiv 2508.00945 ↗First posted 31 Jul 2025cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

Abstract (arXiv)

Vision Language Models (VLMs) face challenges in effectively coordinating diverse attention mechanisms for cross-modal embedding learning, leading to mismatched attention and suboptimal performance. We propose Consistent Cross-layer Regional Alignment (CCRA), which introduces Layer-Patch-wise Cross Attention (LPWCA) to capture fine-grained regional-semantic correlations by jointly weighting patch and layer-wise embedding, and Progressive Attention Integration (PAI) that systematically coordinates LPWCA, layer-wise, and patch-wise attention mechanisms in sequence. This progressive design ensures consistency from semantic to regional levels while preventing attention drift and maximizing individual attention benefits. Experimental results on ten diverse vision-language benchmarks demonstrate that our CCRA-enhanced LLaVA-v1.5-7B model achieves state-of-the-art performance, outperforming all baseline methods with only 3.55M additional parameters, while providing enhanced interpretability through more regionally focused and semantically aligned attention patterns.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 33 of its 44 references that are in the map. First posted on arXiv 31 Jul 2025.

Builds on

33 of 44 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025LaCo: Efficient Layer-wise Compression of Visual Tokens for Multimodal Large Language Models3 Jul
  2. 2023ECoFLaP: Efficient Coarse-to-Fine Layer-Wise Pruning for Vision-Language Models4 Oct

Other references · 31

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  3. 2023Qwen Technical Reportcited by 3,344
  4. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  5. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  6. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  7. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  8. 2019Towards VQA Models That Can Readcited by 1,718
  9. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  10. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  11. 2021Transformers in Vision: A Surveycited by 1,013
  12. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
19 more
  1. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 675
  2. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 625
  3. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  4. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 547
  5. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  6. 2016Dual Attention Networks for Multimodal Reasoning and Matchingcited by 114
  7. 2025HealthGPT: A Medical Large Vision-Language Model for Unifying Comprehension and Generation via Heterogeneous Knowledge Adaptationcited by 114
  8. 2023From CLIP to DINO: Visual Encoders Shout in Multi-modal Large Language Modelscited by 98
  9. 2023LION : Empowering Multimodal Large Language Model with Dual-Level Visual Knowledgecited by 81
  10. 2024What's in the Image? A Deep-Dive into the Vision of Vision Language Modelscited by 68
  11. 2019Are You Looking? Grounding to Multiple Modalities in Vision-and-Language Navigationcited by 55
  12. 2024Transformer Layers as Painterscited by 54
  13. 2024Dense Connector for MLLMscited by 49
  14. 2025EVEv2: Improved Baselines for Encoder-Free Vision-Language Modelscited by 41
  15. 2024MMFuser: Multimodal Multi-Layer Feature Fuser for Fine-Grained Vision-Language Understandingcited by 23
  16. 2025Multimodal Language Models See Better When They Look Shallowercited by 23
  17. 2022A Frustratingly Simple Approach for End-to-End Image Captioningcited by 12
  18. 2024Instruction-Guided Fusion of Multi-Layer Visual Features in Large Vision-Language Modelscited by 7
  19. 2024Peeling Back the Layers: An In-Depth Evaluation of Encoder Architectures in Neural News Recommenderscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025CASA: Cross-Attention over Self-Attention for Efficient Vision-Language Fusion
  2. 2025AttAnchor: Guiding Cross-Modal Token Alignment in VLMs with Attention Anchors
  3. 2025Cross-Layer Vision Smoothing: Enhancing Visual Understanding via Sustained Focus on Key Objects in Large Vision-Language Models
  4. 2022Cross-modal Attention Congruence Regularization for Vision-Language Relation Alignment
  5. 2025Attention Guided Alignment in Efficient Vision-Language Models
  6. 2026Cross-Modal Attention Analysis and Optimization in Vision-Language Models: A Study on Visual Reliability
  7. 2025Bridging Hidden States in Vision-Language Models
  8. 2026From One-to-One to Many-to-Many: Dynamic Cross-Layer Injection for Deep Vision-Language Fusion