mlsanityData to 7 Oct 2026

Do Vision and Language Models Share Concepts? A Vector Space Alignment Study

arXiv 2302.06555 ↗First posted 13 Feb 2023cs.CL cs.AI cs.CVcited by 27

Not on the map. Why?

Abstract (arXiv)

Large-scale pretrained language models (LMs) are said to ``lack the ability to connect utterances to the world'' (Bender and Koller, 2020), because they do not have ``mental models of the world' '(Mitchell and Krakauer, 2023). If so, one would expect LM representations to be unrelated to representations induced by vision models. We present an empirical evaluation across four families of LMs (BERT, GPT-2, OPT and LLaMA-2) and three vision model architectures (ResNet, SegFormer, and MAE). Our experiments show that LMs partially converge towards representations isomorphic to those of vision models, subject to dispersion, polysemy and frequency. This has important implications for both multi-modal processing and the LM understanding debate (Mitchell and Krakauer, 2023).

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Builds on 26 of its 76 references that are in the map. First posted on arXiv 13 Feb 2023.

Builds on

26 of 76 references.

References · 26

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  8. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  10. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  11. 2022Emergent Abilities of Large Language Modelscited by 2,483
  12. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
14 more
  1. 2017Word Translation Without Parallel Datacited by 480
  2. 2018A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddingscited by 145
  3. 2022The Debate Over Understanding in AI's Large Language Modelscited by 142
  4. 2022Linearly Mapping from Image to Text Spacecited by 139
  5. 2019Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)cited by 128
  6. 2021Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Colorcited by 118
  7. 2022Meaning without reference in large language modelscited by 76
  8. 2018On the Limitations of Unsupervised Bilingual Dictionary Inductioncited by 61
  9. 2020Self-supervised learning through the eyes of a childcited by 61
  10. 2018Non-Adversarial Unsupervised Word Translationcited by 19
  11. 2021The Singleton Fallacy: Why Current Critiques of Language Models Miss the Pointcited by 12
  12. 2021Long-range and hierarchical language predictions in brains and algorithmscited by 10
  13. 2017Limitations of Cross-Lingual Learning from Image Searchcited by 2
  14. 2018Why is unsupervised alignment of English embeddings from different algorithms so hard?cited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 3; 2025: 12; 2026: 9.

27 papers.

Sort:

By area:

  1. 2023Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Taskscited by 243
  2. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 161
  3. 2025Harnessing the Universal Geometry of Embeddingscited by 54
  4. 2025Representation Engineering for Large-Language Models: Survey and Research Challengescited by 18
  5. 2025Seeing is Believing, but How Much? A Comprehensive Analysis of Verbalized Calibration in Vision-Language Modelscited by 16
  6. 2026Back into Plato's Cave: Examining Cross-modal Representational Convergence at Scalecited by 10
  7. 2026LatentLens: Revealing Highly Interpretable Visual Tokens in LLMscited by 9
  8. 2023Structural Similarities Between Language Models and Neural Response Measurementscited by 6
  9. 2024Tomato, Tomahto, Tomate: Do Multilingual Language Models Understand Based on Subword-Level Semantic Concepts?cited by 4
  10. 2025Language models align with brain regions that represent concepts across modalitiescited by 4
  11. 2025Model Alignment Searchcited by 3
  12. 2025Seeing What Tastes Good: Revisiting Multimodal Distributional Semantics in the Billion Parameter Eracited by 3
15 more
  1. 2025Re:Verse -- Can Your VLM Read a Manga?cited by 3
  2. 2025Understanding Subword Compositionality of Large Language Modelscited by 3
  3. 2025Research Community Perspectives on "Intelligence" and Large Language Modelscited by 2
  4. 2023Large language models converge toward human-like concept organizationcited by 1
  5. 2024Intriguing Properties of Large Language and Vision Modelscited by 1
  6. 2025Explaining How Visual, Textual and Multimodal Encoders Share Conceptscited by 1
  7. 2026Cross-Modal Taxonomic Generalization in (Vision-) Language Modelscited by 1
  8. 2026SCoCCA: Multi-modal Sparse Concept Decomposition via Canonical Correlation Analysiscited by 1
  9. 2025Empirical Evaluation of Progressive Coding for Sparse Autoencoderscited by 0
  10. 2025What if Othello-Playing Language Models Could See?cited by 0
  11. 2026Alignment among Language, Vision and Action Representationscited by 0
  12. 2026Language-Pretraining-Induced Bias: A Strong Foundation for General Vision Taskscited by 0
  13. 2026Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Modelscited by 0
  14. 2026Convergent Emergence of In-Context Learning Across Modalitiescited by 0
  15. 2026Displacement Geometry Captures Platonic Shared Reality Across Models and Modalitiescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Vision-Language Models Align with Human Neural Representations in Concept Processing
  2. 2024Beyond Sight: Towards Cognitive Alignment in LVLM via Enriched Visual Knowledge
  3. 2025How Visual Representations Map to Language Feature Space in Multimodal LLMs
  4. 2025Interpreting the linear structure of vision-language model embedding spaces
  5. 2022Vision-Language Intelligence: Tasks, Representation Learning, and Large Models
  6. 2025Large Vision-Language Model Alignment and Misalignment: A Survey Through the Lens of Explainability
  7. 2024Unveiling Visual Perception in Language Models: An Attention Head Analysis Approach
  8. 2024Can We Talk Models Into Seeing the World Differently?