# Representations of Text and Images Align From Layer One

arXiv: https://arxiv.org/abs/2601.08017 · HTML: https://mlsanity.com/paper/2601.08017/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 22 of its 35 references that are in the map.
- First posted on arXiv 12 Jan 2026.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM training and visual instruction tuning](https://mlsanity.com/mllm/mllm-training-and-visual-instruction-tuning/)

## Builds on (most cited first, 22 of 22)

- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017)
- [Steering Llama 2 via Contrastive Activation Addition](https://mlsanity.com/paper/2312.06681/) (2023)
- [Steering Language Models With Activation Engineering](https://mlsanity.com/paper/2308.10248/) (2023)
- [Probing Classifiers: Promises, Shortcomings, and Advances](https://mlsanity.com/paper/2102.12452/) (2021)
- [Network Dissection: Quantifying Interpretability of Deep Visual Representations](https://mlsanity.com/paper/1704.05796/) (2017)
- [Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning](https://mlsanity.com/paper/2203.02053/) (2022)
- [The Remarkable Robustness of LLMs: Stages of Inference?](https://mlsanity.com/paper/2406.19384/) (2024)
- [Towards Interpreting Visual Information Processing in Vision-Language Models](https://mlsanity.com/paper/2410.07149/) (2024)
- [Hallucination Augmented Contrastive Learning for Multimodal Large Language Model](https://mlsanity.com/paper/2312.06968/) (2023)
- [Emergent Introspective Awareness in Large Language Models](https://mlsanity.com/paper/2601.01828/) (2026)
- [What's in the Image? A Deep-Dive into the Vision of Vision Language Models](https://mlsanity.com/paper/2411.17491/) (2024)
- [Cross-modal Information Flow in Multimodal Large Language Models](https://mlsanity.com/paper/2411.18620/) (2024)
- [Interpreting CLIP with Hierarchical Sparse Autoencoders](https://mlsanity.com/paper/2502.20578/) (2025)
- [An X-Ray Is Worth 15 Features: Sparse Autoencoders for Interpretable Radiology Report Generation](https://mlsanity.com/paper/2410.03334/) (2024)
- [How Visual Representations Map to Language Feature Space in Multimodal LLMs](https://mlsanity.com/paper/2506.11976/) (2025)
- [Bridge the Modality and Capability Gaps in Vision-Language Model Selection](https://mlsanity.com/paper/2403.13797/) (2024)
- [Ensemble everything everywhere: Multi-scale aggregation for adversarial robustness](https://mlsanity.com/paper/2408.05446/) (2024)
- [Decipher the Modality Gap in Multimodal Contrastive Learning: From Convergent Representations to Pairwise Alignment](https://mlsanity.com/paper/2510.03268/) (2025)
- [Direct Ascent Synthesis: Revealing Hidden Generative Capabilities in Discriminative Models](https://mlsanity.com/paper/2502.07753/) (2025)

## Built on by (most cited first, 3 of 3)

- [LatentLens: Revealing Highly Interpretable Visual Tokens in LLMs](https://mlsanity.com/paper/2602.00462/) (2026)
- [Better Source, Better Flow: Learning Condition-Dependent Source Distribution for Flow Matching](https://mlsanity.com/paper/2602.05951/) (2026)
- [Using OCR Heads to Verbalize Image Semantics](https://mlsanity.com/paper/2609.18823/) (2026)

## Abstract (arXiv)

We show that for a variety of concepts in adapter-based vision-language models, the representations of their images and their text descriptions are meaningfully aligned from the very first layer. This contradicts the established view that such image-text alignment only appears in late layers. We show this using a new synthesis-based method inspired by DeepDream: given a textual concept such as "Jupiter", we extract its concept vector at a given layer, and then use optimisation to synthesise an image whose representation aligns with that vector. We apply our approach to hundreds of concepts across seven layers in Gemma 3, and find that the synthesised images often depict salient visual features of the targeted textual concepts: for example, already at layer 1, more than 50 % of images depict recognisable features of animals, activities, or seasons. Our method thus provides direct, constructive evidence of image-text alignment on a concept-by-concept and layer-by-layer basis. Unlike previous methods for measuring multimodal alignment, our approach is simple, fast, and does not require auxiliary models or datasets. It also offers a new path towards model interpretability, by providing a way to visualise a model's representation space by backtracing through its image processing components.