# Vision Transformers Learn Word Meaning from Images Alone

arXiv: https://arxiv.org/abs/2610.13252 · HTML: https://mlsanity.com/paper/2610.13252/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 30 of its 39 references that are in the map.
- First posted on arXiv 9 Oct 2026.

## Builds on (most cited first, 30 of 30)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017)
- [Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models](https://mlsanity.com/paper/2506.05176/) (2025)
- [SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features](https://mlsanity.com/paper/2502.14786/) (2025)
- [Steering Llama 2 via Contrastive Activation Addition](https://mlsanity.com/paper/2312.06681/) (2023)
- [The Linear Representation Hypothesis and the Geometry of Large Language Models](https://mlsanity.com/paper/2311.03658/) (2023)
- [The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets](https://mlsanity.com/paper/2310.06824/) (2023)
- [FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts](https://mlsanity.com/paper/2311.05608/) (2023)
- [Character Region Awareness for Text Detection](https://mlsanity.com/paper/1904.01941/) (2019)
- [Model evaluation for extreme risks](https://mlsanity.com/paper/2305.15324/) (2023)
- [SimLex-999: Evaluating Semantic Models with (Genuine) Similarity Estimation](https://mlsanity.com/paper/1408.3456/) (2014)
- [What Do Self-Supervised Vision Transformers Learn?](https://mlsanity.com/paper/2305.00729/) (2023)
- [Scaling Language-Free Visual Representation Learning](https://mlsanity.com/paper/2504.01017/) (2025)
- [Disentangling visual and written concepts in CLIP](https://mlsanity.com/paper/2206.07835/) (2022)
- [CLIPPO: Image-and-Language Understanding from Pixels Only](https://mlsanity.com/paper/2212.08045/) (2022)
- [Revisiting the Platonic Representation Hypothesis: An Aristotelian View](https://mlsanity.com/paper/2602.14486/) (2026)
- [Issues in evaluating semantic spaces using word analogies](https://mlsanity.com/paper/1606.07736/) (2016)
- [Parrot Captions Teach CLIP to Spot Text](https://mlsanity.com/paper/2312.14232/) (2023)
- [Back into Plato's Cave: Examining Cross-modal Representational Convergence at Scale](https://mlsanity.com/paper/2604.18572/) (2026)
- [Data or Language Supervision: What Makes CLIP Better than DINO?](https://mlsanity.com/paper/2510.11835/) (2025)
- [Towards an Explainable Comparison and Alignment of Feature Embeddings](https://mlsanity.com/paper/2506.06231/) (2025)
- [SPEECH-COCO: 600k Visually Grounded Spoken Captions Aligned to MSCOCO Data Set](https://mlsanity.com/paper/1707.08435/) (2017)
- [Large Vision Models Can Solve Mental Rotation Problems](https://mlsanity.com/paper/2509.15271/) (2025)

## Abstract (arXiv)

Can a vision model learn what words mean from pixels alone? We show that the two largest DINOv3 vision transformers organize words rendered in images by word meaning, despite being pretrained using self-supervision on images only, without paired captions, class labels, or other language supervision. We derive word representations from embedding differences between word images and matched controls, then test whether semantically related words are neighbors within each model's embedding space, while controlling for spelling similarity. We find that brand names in the same industry group together, as do nouns from the same semantic domain. Country-capital pairs exhibit consistent vector offsets, similarly to word embeddings trained on text. These word representations also align with the representations of the visual object categories they name. DINOv3-H and DINOv3-7B are, to our knowledge, the first self-supervised vision models to show substantial semantic structure for rendered words. Comparisons with text and speech encoders reveal shared structure among representations of the same words. Contrastive alignment with either encoder further strengthened DINOv3-H's word-level semantic structure, while leaving the other finetuned vision models largely unchanged. Image-only training can therefore teach vision models not just what words look like, but also how their meanings relate.