mlsanityData to 14 Oct 2026

The Cost of Language: Centroid Erasure Exposes and Exploits Modal Competition in Multimodal Language Models

arXiv 2604.14363 ↗First posted 15 Apr 2026cs.CL cs.AI cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal language models systematically underperform on visual perception tasks, yet the structure underlying this failure remains poorly understood. We propose centroid replacement, mapping tokens to their nearest K-means centroid and removing within-cluster residual structure, as a controlled probe for modal dependence. Across seven models spanning four architecture families, post-image text replacement costs 4x more accuracy than visual replacement on six BLINK tasks; VPBench and MedBLINK preserve this aggregate ordering. Since text replacement also disrupts the task and answer interface, the gap measures relative dependence. Text centroid contrastive decoding (TCCD) exploits this asymmetry without retraining, recovering up to +16.9% accuracy on an individual task at oracle best-per-task αinterp; every model gains on at least one task, although gains vary across tasks and models. Together, centroid replacement provides a retraining-free audit of modal dependence on labeled data, while TCCD offers a reusable inference-time intervention. Code and artifacts are available at: https://github.com/yahskapar/centroid-erasure

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 44 of its 48 references that are in the map. First posted on arXiv 15 Apr 2026.

Builds on

44 of 48 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Qwen3-VL Technical Report26 Nov
  2. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling6 Dec
  3. 2024LLaVA-OneVision: Easy Visual Task Transfer6 Aug
  4. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs24 Jun
  5. 2024BLINK: Multimodal Large Language Models Can See but Not Perceive18 Apr
  6. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs11 Jan

Other references · 38

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  2. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  3. 2025Qwen2.5-VL Technical Reportcited by 5,587
  4. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  5. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  6. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
  7. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 688
  8. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 642
  9. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 561
  10. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 478
  11. 2025MedGemma Technical Reportcited by 436
  12. 2024The Platonic Representation Hypothesiscited by 382
26 more
  1. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 376
  2. 2024What matters when building vision-language models?cited by 326
  3. 2024VisionZip: Longer is Better but Not Necessary in Vision Language Modelscited by 313
  4. 2022Balanced Multimodal Learning via On-the-fly Gradient Modulationcited by 264
  5. 2024Building and better understanding vision-language models: insights and future directionscited by 173
  6. 2024MDAgents: An Adaptive Collaboration of LLMs for Medical Decision-Makingcited by 163
  7. 2024The Geometry of Categorical and Hierarchical Concepts in Large Language Modelscited by 159
  8. 2025See What You Are Told: Visual Attention Sink in Large Multimodal Modelscited by 132
  9. 2025Words or Vision: Do Vision-Language Models Have Blind Faith in Text?cited by 89
  10. 2024Towards Conversational Diagnostic AIcited by 79
  11. 2026MIRAGE: The Illusion of Visual Understandingcited by 65
  12. 2024MBQ: Modality-Balanced Quantization for Large Vision-Language Modelscited by 45
  13. 2026MedGemma 1.5 Technical Reportcited by 45
  14. 2025Beyond Semantics: Rediscovering Spatial Awareness in Vision-Language Modelscited by 37
  15. 2025Shared Global and Local Geometry of Language Model Embeddingscited by 36
  16. 2025Interpreting the linear structure of vision-language model embedding spacescited by 32
  17. 2024Mitigating Hallucinations in Large Vision-Language Models (LVLMs) via Language-Contrastive Decoding (LCD)cited by 23
  18. 2025To Sink or Not to Sink: Visual Information Pathways in Large Vision-Language Modelscited by 20
  19. 2024Transforming Wearable Data into Personal Health Insights using Large Language Model Agentscited by 19
  20. 2025The Anatomy of a Personal Health Agentcited by 18
  21. 2025Rethinking Multimodal Learning from the Perspective of Mitigating Classification Ability Disproportioncited by 13
  22. 2026VLMs Need Words: Vision Language Models Ignore Visual Detail In Favor of Semantic Anchorscited by 13
  23. 2025The Mirage of Performance Gains: Why Contrastive Decoding Fails to Mitigate Object Hallucinations in MLLMs?cited by 9
  24. 2025Mitigating Modal Imbalance in Multimodal Reasoningcited by 8
  25. 2025Visually Prompted Benchmarks Are Surprisingly Fragilecited by 7
  26. 2026SDCD: Structure-Disrupted Contrastive Decoding for Mitigating Hallucinations in Large Vision-Language Modelscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025VisiPruner: Decoding Discontinuous Cross-Modal Dynamics for Efficient Multimodal LLMs
  2. 2026PerceptionBench: Evaluating Atomic Visual Perception in Multimodal Large Language Models
  3. 2026Towards Understanding Modality Interaction in Multimodal Language Models via Partial Information Decomposition
  4. 2026The Geometry of Compromise: Unlocking Generative Capabilities via Controllable Modality Alignment
  5. 2026Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Models
  6. 2026Mechanistic Insights into Functional Sparsity in Multimodal LLMs via CoRe Heads
  7. 2025Perceiving Beyond Language Priors: Enhancing Visual Comprehension and Attention in Multimodal Models
  8. 2025Demystifying the Visual Quality Paradox in Multimodal Large Language Models