mlsanityData to 14 Oct 2026

Understanding Language Prior of LVLMs by Contrasting Chain-of-Embedding

arXiv 2509.23050 ↗First posted 27 Sep 2025cs.LG cs.AIcited by 14

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Large vision-language models (LVLMs) achieve strong performance on multimodal tasks, yet they often default to their language prior (LP) -- memorized textual patterns from pre-training while under-utilizing visual evidence. Prior analyses of LP mostly rely on input-output probing, which fails to reveal the internal mechanisms governing when and how vision influences model behavior. To address this gap, we present the first systematic analysis of language prior through the lens of chain-of-embedding, which examines the layer-wise representation dynamics within LVLMs. Our analysis reveals a universal phenomenon: each model exhibits a Visual Integration Point (VIP), a critical layer at which visual information begins to meaningfully reshape hidden representations and influence decoding for multimodal reasoning. Building on this observation, we introduce the Total Visual Integration (TVI) estimator, which aggregates representational discrepancy beyond the VIP to quantify how strongly visual query influences response generation. Across 60 model-dataset combinations spanning 10 contemporary LVLMs and 6 benchmarks, we demonstrate that VIP consistently emerges, and that TVI reliably predicts the strength of language prior. This offers a principled toolkit for diagnosing and understanding language prior in LVLMs.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 54 of its 71 references that are in the map. First posted on arXiv 27 Sep 2025.

Builds on

54 of 71 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025How Visual Representations Map to Language Feature Space in Multimodal LLMs13 Jun
  2. 2025Visual Instruction Bottleneck Tuning20 May
  3. 2025Understanding Multimodal LLMs Under Distribution Shifts: An Information-Theoretic Approach1 Feb
  4. 2024Unveiling Visual Perception in Language Models: An Attention Head Analysis Approach24 Dec
  5. 2024LLaVA-OneVision: Easy Visual Task Transfer6 Aug
  6. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs24 Jun
  7. 2024BLINK: Multimodal Large Language Models Can See but Not Perceive18 Apr
  8. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs11 Jan
  9. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  10. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  11. 2023A Survey on Multimodal Large Language Models23 Jun
  12. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
1 more
  1. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 41

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2025Qwen2.5-VL Technical Reportcited by 5,587
  7. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  8. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  9. 2015Domain-Adversarial Training of Neural Networkscited by 3,044
  10. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  11. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  12. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
29 more
  1. 2025Gemma 3 Technical Reportcited by 1,824
  2. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  3. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
  4. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 422
  5. 2025Layer by Layer: Uncovering Hidden Representations in Language Modelscited by 311
  6. 2025SmolVLM: Redefining small and efficient multimodal modelscited by 237
  7. 2024Multi-Modal Hallucination Control by Visual Information Groundingcited by 208
  8. 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMscited by 179
  9. 2024Vision language models are blind: Failing to translate detailed visual features into wordscited by 147
  10. 2024Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lenscited by 143
  11. 2024Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizingcited by 125
  12. 2024Towards Interpreting Visual Information Processing in Vision-Language Modelscited by 122
  13. 2024Not All Layers of LLMs Are Necessary During Inferencecited by 96
  14. 2025Words or Vision: Do Vision-Language Models Have Blind Faith in Text?cited by 89
  15. 2025More Thinking, Less Seeing? Assessing Amplified Hallucination in Multimodal Reasoning Modelscited by 86
  16. 2025Do Multilingual LLMs Think In English?cited by 78
  17. 2024V-DPO: Mitigating Hallucination in Large Vision Language Models via Vision-Guided Direct Preference Optimizationcited by 67
  18. 2025Vision Language Models are Biasedcited by 66
  19. 2025Eagle 2.5: Boosting Long-Context Post-Training for Frontier Vision-Language Modelscited by 64
  20. 2024VLind-Bench: Measuring Language Priors in Large Vision-Language Modelscited by 45
  21. 2024Probing Visual Language Priors in VLMscited by 45
  22. 2023Improving Language Plasticity via Pretraining with Active Forgettingcited by 43
  23. 2024On Large Language Models' Hallucination with Regard to Known Factscited by 43
  24. 2023Towards Calibrated Robust Fine-Tuning of Vision-Language Modelscited by 35
  25. 2024Attend First, Consolidate Later: On the Importance of Attention in Different LLM Layerscited by 25
  26. 2024Mitigating the Linguistic Gap with Phonemic Representations for Robust Cross-lingual Transfercited by 6
  27. 2025CoRE: Enhancing Metacognition with Label-free Self-evaluation in LRMscited by 5
  28. 2024Layer-wise Alignment: Examining Safety Alignment Across Image Encoder Layers in Vision Language Modelscited by 4
  29. 2025Prioritizing Image-Related Tokens Enhances Vision-Language Pre-Trainingcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 14.

14 papers.

Sort:

By area:

  1. 2026Do VLMs Perceive or Recall? Probing Visual Perception vs. Memory with Classic Visual Illusionscited by 8
  2. 2026Staying VIGILant: Mitigating Visual Laziness via Counterfactual Visual Alignment in MLLMscited by 8
  3. 2026Where Reliability Lives in Vision-Language Models: A Mechanistic Study of Attention, Hidden States, and Causal Circuitscited by 3
  4. 2026When Does a Video-Language Model Stop Watching? Reward Strength Controls the Formation and Reversal of Visual Shortcuts in Multimodal RLVRcited by 2
  5. 2026Boosting Visual Instruction Tuning with Self-Supervised Guidancecited by 1
  6. 2026From Accuracy to Visual Dependence: Auditing and Filtering Modality Collapse in Traffic VideoQAcited by 1
  7. 2026HoloCount: A Holistic Visual Counting Benchmark for MLLMscited by 1
  8. 2026Leveraging Visual Signals for Robust Token-Level Uncertainty in Vision-Language Generationcited by 0
  9. 2026Visuals Lie, Consistency Speaks: Disentangling Spatial Attention from Reliability in Vision-Language Modelscited by 0
  10. 2026Look Clearly Before Answering: Mitigating Hallucinations in LVLMs via Saliency-Driven Perceptual Realignmentcited by 0
  11. 2026Confidence Estimation for Financial Vision-Language Models in Chart and Document Understandingcited by 0
  12. 2026(How) Do MLLMs Report Bistable Images Like Humans?cited by 0
2 more
  1. 2026CCRV-Bench: Constraint-Based Evaluation of Causal Reasoning in Vision-Language Modelscited by 0
  2. 2026Learning How to Remember: Memory Modality Selection for GUI Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Perceiving Beyond Language Priors: Enhancing Visual Comprehension and Attention in Multimodal Models
  2. 2026The Hidden Evolution of Disguised Visual Context inside the VLM
  3. 2025Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training
  4. 2026A Comprehensive Information-Decomposition Analysis of Large Vision-Language Models
  5. 2026Predictive Regularization Against Visual Representation Degradation in Multimodal Large Language Models
  6. 2026Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Models
  7. 2026Beyond Language Modeling: An Exploration of Multimodal Pretraining
  8. 2025How Multimodal LLMs Solve Image Tasks: A Lens on Visual Grounding, Task Reasoning, and Answer Decoding