mlsanityData to 7 Oct 2026

The Alignment Illusion in Multimodal Large Language Models

arXiv 2609.30210 ↗First posted 24 Sep 2026cs.CV cs.LGcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Layer-wise visual-text similarity in Multimodal Large Language Models (MLLMs) is widely interpreted as evidence that the language model progressively integrates visual content into a shared representation space. This reading rests on the assumption that scalar alignment scores reflect content-level cross-modal interaction. To test this assumption, we apply controlled interventions to the visual stream. Across 13 MLLMs from five families spanning 0.5B to 72B parameters, replacing projector-output visual tokens with Gaussian noise sharply reduces task accuracy, yet four standard scalar measures (CKA, SVCCA, MIR, and the leading principal-angle cosine) fail to consistently separate the corrupted stream from the original. We call this failure the alignment illusion and trace it to the shared language-model pathway: anisotropic MLP down-projections pull visual and text tokens toward common output directions, producing weight-induced alignment. Because this component is essentially one-dimensional, we introduce the principal-angle gap (PA gap), defined as the difference between the top two principal-angle cosines, which separates weight-induced similarity from multi-directional visual structure. Under graded visual corruption, the PA gap tracks task accuracy more consistently than the scalar scores we consider; under a structured but irrelevant image, it further exposes regimes in which internal geometry and task accuracy come apart. Internal visual-text alignment in MLLMs is therefore best read as a geometric diagnostic of the visual stream inside the language model rather than a direct proxy for content-level cross-modal interaction, and is most informative when calibrated by controlled task evidence.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 20 of its 24 references that are in the map. First posted on arXiv 24 Sep 2026.

Builds on

20 of 24 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Vision-Language Models Create Cross-Modal Task Representations29 Oct
  2. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  3. 2024LLaVA-OneVision: Easy Visual Task Transfer6 Aug
  4. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs11 Jan
  5. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul

Other references · 15

  1. 2023Visual Instruction Tuningcited by 7,792
  2. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  3. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  4. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  5. 2018Sanity Checks for Saliency Mapscited by 1,149
  6. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  7. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 411
  8. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 374
  9. 2021All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Qualitycited by 150
  10. 2024Towards Interpreting Visual Information Processing in Vision-Language Modelscited by 115
  11. 2022Reliability of CKA as a Similarity Measure in Deep Learningcited by 82
  12. 2023Towards Vision-Language Mechanistic Interpretability: A Causal Tracing Tool for BLIPcited by 62
3 more
  1. 2024Understanding Information Storage and Transfer in Multi-modal Large Language Modelscited by 48
  2. 2025Understanding the Emergence of Multimodal Representation Alignmentcited by 28
  3. 2024Losing Visual Needles in Image Haystacks: Vision Language Models are Easily Distracted in Short and Long Contextscited by 14

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Can Linguistic Knowledge Improve Multimodal Alignment in Vision-Language Pretraining?
  2. 2026Fill the GAP: A Granular Alignment Paradigm for Visual Reasoning in Multimodal Large Language Models
  3. 2025Visual Representation Alignment for Multimodal Large Language Models
  4. 2025Perceiving Beyond Language Priors: Enhancing Visual Comprehension and Attention in Multimodal Models
  5. 2024Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputs
  6. 2025The Unseen Bias: How Norm Discrepancy in Pre-Norm MLLMs Leads to Visual Information Loss
  7. 2026Staying VIGILant: Mitigating Visual Laziness via Counterfactual Visual Alignment in MLLMs
  8. 2026Modality Gap-Driven Subspace Alignment Training Paradigm For Multimodal Large Language Models