mlsanityData to 14 Oct 2026

A quantitative analysis of semantic information in deep representations of text and images

arXiv 2505.17101 ↗First posted 21 May 2025cs.CL cs.LG physics.comp-phcited by 7

Not on the map. Why?

Abstract (arXiv)

It was recently observed that the representations of different models that process identical or semantically related inputs tend to align. We analyze this phenomenon using the Information Imbalance, an asymmetric rank-based measure that quantifies the capability of a representation to predict another, providing a proxy of the cross-entropy which can be computed efficiently in high-dimensional spaces. By measuring the Information Imbalance between representations generated by DeepSeek-V3 processing translations, we find that semantic information is spread across many tokens, and that semantic predictability is strongest in a set of central layers of the network, robust across six language pairs. We measure clear information asymmetries: English representations are systematically more predictive than those of other languages, and DeepSeek-V3 representations are more predictive of those in a smaller model such as Llama3-8b than the opposite. In the visual domain, we observe that semantic information concentrates in middle layers for autoregressive models and in final layers for encoder models, and these same layers yield the strongest cross-modal predictability with textual representations of image captions. Our results support the hypothesis of semantic convergence across languages, modalities, and architectures, while showing that directed predictability between representations varies strongly with layer-depth, model scale, and language.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 37 of its 54 references that are in the map. First posted on arXiv 21 May 2025.

Builds on

37 of 54 references.

References · 37

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  3. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  4. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  5. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 834
  6. 2016Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activationscited by 426
  7. 2025Layer by Layer: Uncovering Hidden Representations in Language Modelscited by 311
  8. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 295
  9. 2024Massive Activations in Large Language Modelscited by 294
  10. 2024Do Llamas Work in English? On the Latent Language of Multilingual Transformerscited by 281
  11. 2014Understanding image representations by measuring their equivariance and equivalencecited by 243
  12. 2021Revisiting Model Stitching to Compare Neural Representationscited by 209
25 more
  1. 2022Relative representations enable zero-shot latent space communicationcited by 191
  2. 2023BitNet: Scaling 1-bit Transformers for Large Language Modelscited by 189
  3. 2023Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothingcited by 151
  4. 2024How do Large Language Models Handle Multilingualism?cited by 150
  5. 2023A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operationscited by 144
  6. 2021BERT Busters: Outlier Dimensions that Disrupt Transformerscited by 112
  7. 2023The geometry of hidden representations of large transformer modelscited by 112
  8. 2024EuroLLM: Multilingual Language Models for Europecited by 106
  9. 2018A Survey on Methods and Theories of Quantized Neural Networkscited by 75
  10. 2025Large Language Models Share Representations of Latent Grammatical Concepts Across Typologically Diverse Languagescited by 59
  11. 2024Emergence of a High-Dimensional Abstraction Phase in Language Transformerscited by 56
  12. 2022ASIF: Coupled Data Turns Unimodal Models to Multimodal Without Trainingcited by 47
  13. 2024Characterizing Truthfulness in Large Language Model Generations with Local Intrinsic Dimensioncited by 45
  14. 2024Do Vision and Language Encoders Represent the World Similarly?cited by 44
  15. 2020Language Through a Prism: A Spectral Approach for Multiscale Language Representationscited by 39
  16. 2023Large language models implicitly learn to straighten neural sentence trajectories to construct a predictive representation of natural languagecited by 32
  17. 2020Hierarchical nucleation in deep neural networkscited by 31
  18. 2025Middle-Layer Representation Alignment for Cross-Lingual Transfer in Fine-Tuned LLMscited by 29
  19. 2017The High-Dimensional Geometry of Binary Neural Networkscited by 22
  20. 2021Ranking the information content of distance measurescited by 22
  21. 2024Multilingual Large Language Models: A Systematic Surveycited by 18
  22. 2024Evidence from fMRI Supports a Two-Phase Abstraction Process in Language Modelscited by 13
  23. 2024Automatic feature selection and weighting in molecular systems using Differentiable Information Imbalancecited by 5
  24. 2024Unsupervised detection of semantic correlations in big datacited by 2
  25. 2025Linear scaling causal discovery from high-dimensional time series by dynamical community detectioncited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 7.

7 papers.

  1. 2026Differential syntactic and semantic encoding in LLMscited by 7
  2. 2026Tracing the complexity profiles of different linguistic phenomena through the intrinsic dimension of LLM representationscited by 3
  3. 2026Abstraction Induces the Brain Alignment of Language and Speech Modelscited by 2
  4. 2026Signal-to-Noise Ratio and Sample Size Govern Representational Alignment in Neural Networkscited by 0
  5. 2026Visual Instruction Tuning Aligns Modalities through Abstractioncited by 0
  6. 2026The Platonic Defense: Backdoor Defense for Self-Supervised Encoders in the Era of Large Scale Pre-trainingcited by 0
  7. 2026Texture Representations in Deep Vision Models: Comparing CNNs, Vision Transformers, and Human Perceptioncited by 0