# Do Vision and Language Models Share Concepts? A Vector Space Alignment Study

arXiv: https://arxiv.org/abs/2302.06555 · HTML: https://mlsanity.com/paper/2302.06555/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 76 references that are in the map.
- First posted on arXiv 13 Feb 2023.

## Builds on (most cited first, 26 of 26)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [Word Translation Without Parallel Data](https://mlsanity.com/paper/1710.04087/) (2017)
- [A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings](https://mlsanity.com/paper/1805.06297/) (2018)
- [Linearly Mapping from Image to Text Space](https://mlsanity.com/paper/2209.15162/) (2022)
- [The Debate Over Understanding in AI's Large Language Models](https://mlsanity.com/paper/2210.13966/) (2022)
- [Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)](https://mlsanity.com/paper/1905.11833/) (2019)
- [Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Color](https://mlsanity.com/paper/2109.06129/) (2021)
- [Meaning without reference in large language models](https://mlsanity.com/paper/2208.02957/) (2022)
- [On the Limitations of Unsupervised Bilingual Dictionary Induction](https://mlsanity.com/paper/1805.03620/) (2018)
- [Self-supervised learning through the eyes of a child](https://mlsanity.com/paper/2007.16189/) (2020)
- [Non-Adversarial Unsupervised Word Translation](https://mlsanity.com/paper/1801.06126/) (2018)
- [The Singleton Fallacy: Why Current Critiques of Language Models Miss the Point](https://mlsanity.com/paper/2102.04310/) (2021)
- [Long-range and hierarchical language predictions in brains and algorithms](https://mlsanity.com/paper/2111.14232/) (2021)
- [Limitations of Cross-Lingual Learning from Image Search](https://mlsanity.com/paper/1709.05914/) (2017)
- [Why is unsupervised alignment of English embeddings from different algorithms so hard?](https://mlsanity.com/paper/1809.00150/) (2018)

## Built on by (most cited first, 27 of 27)

- [Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks](https://mlsanity.com/paper/2307.02477/) (2023)
- [Understanding World or Predicting Future? A Comprehensive Survey of World Models](https://mlsanity.com/paper/2411.14499/) (2024)
- [Harnessing the Universal Geometry of Embeddings](https://mlsanity.com/paper/2505.12540/) (2025)
- [Representation Engineering for Large-Language Models: Survey and Research Challenges](https://mlsanity.com/paper/2502.17601/) (2025)
- [Seeing is Believing, but How Much? A Comprehensive Analysis of Verbalized Calibration in Vision-Language Models](https://mlsanity.com/paper/2505.20236/) (2025)
- [Back into Plato's Cave: Examining Cross-modal Representational Convergence at Scale](https://mlsanity.com/paper/2604.18572/) (2026)
- [LatentLens: Revealing Highly Interpretable Visual Tokens in LLMs](https://mlsanity.com/paper/2602.00462/) (2026)
- [Structural Similarities Between Language Models and Neural Response Measurements](https://mlsanity.com/paper/2306.01930/) (2023)
- [Tomato, Tomahto, Tomate: Do Multilingual Language Models Understand Based on Subword-Level Semantic Concepts?](https://mlsanity.com/paper/2411.04530/) (2024)
- [Language models align with brain regions that represent concepts across modalities](https://mlsanity.com/paper/2508.11536/) (2025)
- [Model Alignment Search](https://mlsanity.com/paper/2501.06164/) (2025)
- [Seeing What Tastes Good: Revisiting Multimodal Distributional Semantics in the Billion Parameter Era](https://mlsanity.com/paper/2506.03994/) (2025)
- [Re:Verse -- Can Your VLM Read a Manga?](https://mlsanity.com/paper/2508.08508/) (2025)
- [Understanding Subword Compositionality of Large Language Models](https://mlsanity.com/paper/2508.17953/) (2025)
- [Research Community Perspectives on "Intelligence" and Large Language Models](https://mlsanity.com/paper/2505.20959/) (2025)
- [Large language models converge toward human-like concept organization](https://mlsanity.com/paper/2308.15047/) (2023)
- [Intriguing Properties of Large Language and Vision Models](https://mlsanity.com/paper/2410.04751/) (2024)
- [Explaining How Visual, Textual and Multimodal Encoders Share Concepts](https://mlsanity.com/paper/2507.18512/) (2025)
- [Cross-Modal Taxonomic Generalization in (Vision-) Language Models](https://mlsanity.com/paper/2603.07474/) (2026)
- [SCoCCA: Multi-modal Sparse Concept Decomposition via Canonical Correlation Analysis](https://mlsanity.com/paper/2603.13884/) (2026)
- [Language-Pretraining-Induced Bias: A Strong Foundation for General Vision Tasks](https://mlsanity.com/paper/2604.01833/) (2026)
- [Convergent Emergence of In-Context Learning Across Modalities](https://mlsanity.com/paper/2609.14011/) (2026)
- [Empirical Evaluation of Progressive Coding for Sparse Autoencoders](https://mlsanity.com/paper/2505.00190/) (2025)
- [What if Othello-Playing Language Models Could See?](https://mlsanity.com/paper/2507.14520/) (2025)
- [Alignment among Language, Vision and Action Representations](https://mlsanity.com/paper/2601.22948/) (2026)
- [Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Models](https://mlsanity.com/paper/2607.26326/) (2026)
- [Displacement Geometry Captures Platonic Shared Reality Across Models and Modalities](https://mlsanity.com/paper/2609.24209/) (2026)

## Abstract (arXiv)

Large-scale pretrained language models (LMs) are said to ``lack the ability to connect utterances to the world'' (Bender and Koller, 2020), because they do not have ``mental models of the world' '(Mitchell and Krakauer, 2023). If so, one would expect LM representations to be unrelated to representations induced by vision models. We present an empirical evaluation across four families of LMs (BERT, GPT-2, OPT and LLaMA-2) and three vision model architectures (ResNet, SegFormer, and MAE). Our experiments show that LMs partially converge towards representations isomorphic to those of vision models, subject to dispersion, polysemy and frequency. This has important implications for both multi-modal processing and the LM understanding debate (Mitchell and Krakauer, 2023).