mlsanityData to 7 Oct 2026

Probing Multimodal Large Language Models for Global and Local Semantic Representations

arXiv 2402.17304 ↗First posted 27 Feb 2024cs.CL cs.AIcited by 12

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

The advancement of Multimodal Large Language Models (MLLMs) has greatly accelerated the development of applications in understanding integrated texts and images. Recent works leverage image-caption datasets to train MLLMs, achieving state-of-the-art performance on image-to-text tasks. However, there are few studies exploring which layers of MLLMs make the most effort to the global image information, which plays vital roles in multimodal comprehension and generation. In this study, we find that the intermediate layers of models can encode more global semantic information, whose representation vectors perform better on visual-language entailment tasks, rather than the topmost layers. We further probe models regarding local semantic representations through object recognition tasks. We find that the topmost layers may excessively focus on local information, leading to a diminished ability to encode global information. Our code and data are released via https://github.com/kobayashikanna01/probing_MLLM_rep.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 27 of its 41 references that are in the map. First posted on arXiv 27 Feb 2024.

Builds on

27 of 41 references.

References · 27

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  8. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  9. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  10. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  11. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  12. 2020A Primer in BERTology: What we know about how BERT workscited by 807
15 more
  1. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 436
  2. 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 382
  3. 2019Revealing the Dark Secrets of BERTcited by 300
  4. 2019Visual Entailment: A Novel Task for Fine-Grained Image Understandingcited by 250
  5. 2019oLMpics -- On what Language Model Pre-training Capturescited by 161
  6. 2020Probing Pretrained Language Models for Lexical Semanticscited by 156
  7. 2020On the Effect of Dropping Layers of Pre-trained Transformer Modelscited by 118
  8. 2020Finding Universal Grammatical Relations in Multilingual BERTcited by 108
  9. 2023Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenizationcited by 87
  10. 2022Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-trainingcited by 71
  11. 2021Discourse Probing of Pretrained Language Modelscited by 38
  12. 2021Probing Multimodal Embeddings for Linguistic Properties: the Visual-Semantic Casecited by 20
  13. 2017Compare, Compress and Propagate: Enhancing Neural Architectures with Alignment Factorization for Natural Language Inferencecited by 10
  14. 2022Probing Cross-modal Semantics Alignment Capability from the Textual Perspectivecited by 3
  15. 2022Fuse Local and Global Semantics in Representation Learningcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 5; 2026: 3.

12 papers.

  1. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  2. 2025A Survey on Mechanistic Interpretability for Multi-Modal Foundation Modelscited by 34
  3. 2024Have the VLMs Lost Confidence? A Study of Sycophancy in VLMscited by 15
  4. 2024Performance Gap in Entity Knowledge Extraction Across Modalities in Vision Language Modelscited by 12
  5. 2025I-FailSense: Towards General Robotic Failure Detection with Vision-Language Modelscited by 11
  6. 2024A Review of Multimodal Explainable Artificial Intelligence: Past, Present and Futurecited by 9
  7. 2025LongAnimation: Long Animation Generation with Dynamic Global-Local Memorycited by 6
  8. 2025RCI: A Score for Evaluating Global and Local Reasoning in Multimodal Benchmarkscited by 2
  9. 2025MIMIC: Multimodal Inversion for Model Interpretation and Conceptualizationcited by 1
  10. 2026PREGEN: Uncovering Latent Thoughts in Composed Video Retrievalcited by 0
  11. 2026MIRROR: Aligning Semantic Relations from Language to Image via Gromov--Wassersteincited by 0
  12. 2026Vision-language models know more about agriculture than they show and rubric-grounded verifications close the gapcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Multimodal Language Models See Better When They Look Shallower
  2. 2024Incorporating Visual Experts to Resolve the Information Loss in Multimodal Large Language Models
  3. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
  4. 2024E5-V: Universal Embeddings with Multimodal Large Language Models
  5. 2024The Revolution of Multimodal Large Language Models: A Survey
  6. 2025How Multimodal LLMs Solve Image Tasks: A Lens on Visual Grounding, Task Reasoning, and Answer Decoding
  7. 2025Multimodal Large Language Models for Text-rich Image Understanding: A Comprehensive Review
  8. 2026Caption First, VQA Second: Knowledge Density, Not Task Format, Drives Multimodal Scaling