mlsanityData to 14 Oct 2026

Cross-Modal Attention Analysis and Optimization in Vision-Language Models: A Study on Visual Reliability

arXiv 2604.17217 ↗First posted 19 Apr 2026cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Vision-Language Models (VLMs) achieve strong cross-modal performance, yet recent evidence suggests they over-rely on textual descriptions while under-utilizing visual evidence -- a phenomenon termed ``text shortcut learning.'' We propose an adversarial evaluation framework that quantifies this cross-modal dependency by measuring accuracy degradation (Drop) when semantically conflicting text is paired with unchanged images. Four adversarial strategies -- shapeswap, colorswap, positionswap, and randomtext -- are applied to a controlled geometric-shapes dataset (n=1,000). We compare three configurations: Baseline CLIP (ViT-B/32), LoRA fine-tuning, and LoRA Optimized (integrating Hard Negative Mining, Label Smoothing, layer-wise learning rates, Cosine Restarts, curriculum learning, and data augmentation). The optimized model reduces average Drop from 27.5% to 9.8% (64.4% relative improvement, p<0.001) while maintaining 97% normal accuracy. Attention visualization and embedding-space analysis confirm that the optimized model attends more to visual features and achieves tighter cross-modal alignment.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 10 of its 12 references that are in the map. First posted on arXiv 19 Apr 2026.

Builds on

10 of 12 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Debiasing Multimodal Large Language Models via Penalization of Language Priors8 Mar

Other references · 9

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023Visual Instruction Tuningcited by 7,869
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  5. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  6. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 561
  7. 2017Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answeringcited by 339
  8. 2021VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomenacited by 135
  9. 2025Words or Vision: Do Vision-Language Models Have Blind Faith in Text?cited by 89

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026One Modality to Forget Them All: Enhancing Cross-Modal Unlearning in Vision-Language Models
  2. 2025Beyond Cross-Modal Alignment: Measuring and Leveraging Modality Gap in Vision-Language Models
  3. 2026Do Vision-Language Models Truly Perform Vision Reasoning? A Rigorous Study of the Modality Gap
  4. 2024Cross-Modal Consistency in Multimodal Large Language Models
  5. 2025Mixed Signals: Decoding VLMs' Reasoning and Underlying Bias in Vision-Language Conflict
  6. 2025Same Content, Different Answers: Cross-Modal Inconsistency in MLLMs
  7. 2025Rethinking Visual Token Reduction in LVLMs Under Cross-Modal Misalignment
  8. 2025Same Task, Different Circuits: Disentangling Modality-Specific Mechanisms in VLMs