mlsanityData to 13 Oct 2026

A Comprehensive Information-Decomposition Analysis of Large Vision-Language Models

arXiv 2603.29676 ↗First posted 31 Mar 2026cs.LG cs.CL cs.CVcited by 4

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Large vision-language models (LVLMs) achieve impressive performance, yet their internal decision-making processes remain opaque, making it difficult to determine if the success stems from true multimodal fusion or from reliance on unimodal priors. To address this attribution gap, we introduce a novel framework using partial information decomposition (PID) to quantitatively measure the "information spectrum" of LVLMs -- decomposing a model's decision-relevant information into redundant, unique, and synergistic components. By adapting a scalable estimator to modern LVLM outputs, our model-agnostic pipeline profiles 26 LVLMs on four datasets across three dimensions -- breadth (cross-model & cross-task), depth (layer-wise information dynamics), and time (learning dynamics across training). Our analysis reveals two key results: (i) two task regimes (synergy-driven vs. knowledge-driven) and (ii) two stable, contrasting family-level strategies (fusion-centric vs. language-centric). We also uncover a consistent three-phase pattern in layer-wise processing and identify visual instruction tuning as the key stage where fusion is learned. Together, these contributions provide a quantitative lens beyond accuracy-only evaluation and offer insights for analyzing and designing the next generation of LVLMs. Code and data are available at https://github.com/RiiShin/pid-lvlm-analysis .

Key facts. Cited by 4 papers in the map (data through 13 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 62 of its 75 references that are in the map. First posted on arXiv 31 Mar 2026.

Builds on

62 of 75 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Same Task, Different Circuits: Disentangling Modality-Specific Mechanisms in VLMs10 Jun
  2. 2024Cross-modal Information Flow in Multimodal Large Language Models27 Nov
  3. 2024Towards Interpreting Visual Information Processing in Vision-Language Models9 Oct
  4. 2024Understanding Information Storage and Transfer in Multi-modal Large Language Models6 Jun
  5. 2023Visual Explanations of Image-Text Representations via Multi-Modal Information Bottleneck Attribution28 Dec
  6. 2022Correlation Information Bottleneck: Towards Adapting Pretrained Multimodal Models for Robust Visual Question Answering14 Sep
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr

Other references · 55

  1. 2017Attention Is All You Needcited by 60,106
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,550
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,549
  5. 2024The Llama 3 Herd of Modelscited by 14,486
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,072
  7. 2023Visual Instruction Tuningcited by 7,827
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,103
  9. 2025Qwen2.5-VL Technical Reportcited by 5,499
  10. 2023Improved Baselines with Visual Instruction Tuningcited by 4,389
  11. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,191
  12. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,855
38 more
  1. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,864
  2. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,692
  3. 2022Locating and Editing Factual Associations in GPTcited by 2,650
  4. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,295
  5. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,118
  6. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,798
  7. 2025Gemma 3 Technical Reportcited by 1,748
  8. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,709
  9. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,586
  10. 2016Understanding intermediate layers using linear classifier probescited by 1,313
  11. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,049
  12. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,009
  13. 2019Attention is not Explanationcited by 922
  14. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 827
  15. 2020Quantifying Attention Flow in Transformerscited by 767
  16. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 711
  17. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 659
  18. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 600
  19. 2023PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answeringcited by 334
  20. 2021Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformerscited by 318
  21. 2023Interpreting CLIP's Image Representation via Text-Based Decompositioncited by 194
  22. 2022Linearly Mapping from Image to Text Spacecited by 141
  23. 2025See What You Are Told: Visual Attention Sink in Large Multimodal Modelscited by 127
  24. 2024NVLM: Open Frontier-Class Multimodal LLMscited by 125
  25. 2020Restricting the Flow: Information Bottlenecks for Attributioncited by 110
  26. 2024Interpreting and Editing Vision-Language Representations to Mitigate Hallucinationscited by 104
  27. 2023Quantifying & Modeling Multimodal Interactions: An Information Decomposition Frameworkcited by 81
  28. 2023Multimodal Neurons in Pretrained Text-Only Transformerscited by 49
  29. 2024Reefknot: A Comprehensive Benchmark for Relation Hallucination Evaluation, Analysis and Mitigation in Multimodal Large Language Modelscited by 30
  30. 2023Multimodal Learning Without Labeled Multimodal Data: Guarantees and Applicationscited by 21
  31. 2025Towards eliciting latent knowledge from LLMs with mechanistic interpretabilitycited by 19
  32. 2024Neuro-Inspired Information-Theoretic Hierarchical Perception for Multimodal Learningcited by 18
  33. 2024Law of Vision Representation in MLLMscited by 17
  34. 2025Understanding Multimodal LLMs Under Distribution Shifts: An Information-Theoretic Approachcited by 15
  35. 2025Learning Optimal Multimodal Information Bottleneck Representationscited by 15
  36. 2025Aligning Multimodal Representations through an Information Bottleneckcited by 13
  37. 2024Quantifying Knowledge Distillation Using Partial Information Decompositioncited by 12
  38. 2022A Measure of the Complexity of Neural Representations based on Partial Information Decompositioncited by 10

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Towards Understanding Modality Interaction in Multimodal Language Models via Partial Information Decompositioncited by 5
  2. 2026Mitigating Object Hallucination in Large Vision-Language Models via False Discovery Controlled Visual Data Splittingcited by 0
  3. 2026Interpreting Reasoning of Large Language Models via Partial Information Decompositioncited by 0
  4. 2026GeoPID: Decomposing and Steering Visual Information in Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Large Vision-Language Models Get Lost in Attention
  2. 2025Understanding Language Prior of LVLMs by Contrasting Chain-of-Embedding
  3. 2026Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospects
  4. 2026Benchmarking Deflection and Hallucination in Large Vision-Language Models
  5. 2026How Vision Becomes Language: A Layer-wise Information-Theoretic Analysis of Multimodal Reasoning
  6. 2023ReForm-Eval: Evaluating Large Vision Language Models via Unified Re-Formulation of Task-Oriented Benchmarks
  7. 2026Not All Error Yields to Scale: Where Scaling Stops in Vision-Language Inference
  8. 2024Instruction-Guided Fusion of Multi-Layer Visual Features in Large Vision-Language Models