mlsanityData to 7 Oct 2026

Representations of Text and Images Align From Layer One

arXiv 2601.08017 ↗First posted 12 Jan 2026cs.CV cs.AIcited by 3

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

We show that for a variety of concepts in adapter-based vision-language models, the representations of their images and their text descriptions are meaningfully aligned from the very first layer. This contradicts the established view that such image-text alignment only appears in late layers. We show this using a new synthesis-based method inspired by DeepDream: given a textual concept such as "Jupiter", we extract its concept vector at a given layer, and then use optimisation to synthesise an image whose representation aligns with that vector. We apply our approach to hundreds of concepts across seven layers in Gemma 3, and find that the synthesised images often depict salient visual features of the targeted textual concepts: for example, already at layer 1, more than 50 % of images depict recognisable features of animals, activities, or seasons. Our method thus provides direct, constructive evidence of image-text alignment on a concept-by-concept and layer-by-layer basis. Unlike previous methods for measuring multimodal alignment, our approach is simple, fast, and does not require auxiliary models or datasets. It also offers a new path towards model interpretability, by providing a way to visualise a model's representation space by backtracing through its image processing components.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 22 of its 35 references that are in the map. First posted on arXiv 12 Jan 2026.

Builds on

22 of 35 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025How Visual Representations Map to Language Feature Space in Multimodal LLMs13 Jun

Other references · 21

  1. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  2. 2025Gemma 3 Technical Reportcited by 1,686
  3. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  4. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,309
  5. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,009
  6. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
  7. 2023Steering Language Models With Activation Engineeringcited by 824
  8. 2017Network Dissection: Quantifying Interpretability of Deep Visual Representationscited by 703
  9. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 553
  10. 2024The Remarkable Robustness of LLMs: Stages of Inference?cited by 138
  11. 2023Hallucination Augmented Contrastive Learning for Multimodal Large Language Modelcited by 119
  12. 2024Towards Interpreting Visual Information Processing in Vision-Language Modelscited by 115
9 more
  1. 2026Emergent Introspective Awareness in Large Language Modelscited by 73
  2. 2024What's in the Image? A Deep-Dive into the Vision of Vision Language Modelscited by 66
  3. 2024Cross-modal Information Flow in Multimodal Large Language Modelscited by 65
  4. 2025Interpreting CLIP with Hierarchical Sparse Autoencoderscited by 54
  5. 2024An X-Ray Is Worth 15 Features: Sparse Autoencoders for Interpretable Radiology Report Generationcited by 23
  6. 2024Bridge the Modality and Capability Gaps in Vision-Language Model Selectioncited by 20
  7. 2024Ensemble everything everywhere: Multi-scale aggregation for adversarial robustnesscited by 12
  8. 2025Decipher the Modality Gap in Multimodal Contrastive Learning: From Convergent Representations to Pairwise Alignmentcited by 10
  9. 2025Direct Ascent Synthesis: Revealing Hidden Generative Capabilities in Discriminative Modelscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026LatentLens: Revealing Highly Interpretable Visual Tokens in LLMscited by 9
  2. 2026Better Source, Better Flow: Learning Condition-Dependent Source Distribution for Flow Matchingcited by 8
  3. 2026Using OCR Heads to Verbalize Image Semanticscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023What You See is What You Read? Improving Text-Image Alignment Evaluation
  2. 2024ComAlign: Compositional Alignment in Vision-Language Models
  3. 2025Large Vision-Language Model Alignment and Misalignment: A Survey Through the Lens of Explainability
  4. 2023Improving Compositional Text-to-image Generation with Large Vision-Language Models
  5. 2025AlignVLM: Bridging Vision and Language Latent Spaces for Multimodal Document Understanding
  6. 2024Seeing the Image: Prioritizing Visual Correlation by Contrastive Alignment
  7. 2024SEA: Supervised Embedding Alignment for Token-Level Visual-Textual Integration in MLLMs
  8. 2025Visual Representation Alignment for Multimodal Large Language Models