# Relative representations enable zero-shot latent space communication

arXiv: https://arxiv.org/abs/2209.15430 · HTML: https://mlsanity.com/paper/2209.15430/

## Key facts

- Cited by 191 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 56 references that are in the map.
- First posted on arXiv 30 Sep 2022.

## Builds on (most cited first, 30 of 36)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Fast Graph Representation Learning with PyTorch Geometric](https://mlsanity.com/paper/1903.02428/) (2019)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Instance Normalization: The Missing Ingredient for Fast Stylization](https://mlsanity.com/paper/1607.08022/) (2016)
- [Revisiting Semi-Supervised Learning with Graph Embeddings](https://mlsanity.com/paper/1603.08861/) (2016)
- [The Implicit Bias of Gradient Descent on Separable Data](https://mlsanity.com/paper/1710.10345/) (2017)
- [Datasets: A Community Library for Natural Language Processing](https://mlsanity.com/paper/2109.02846/) (2021)
- [Word Translation Without Parallel Data](https://mlsanity.com/paper/1710.04087/) (2017)
- [Insights on representational similarity in neural networks with canonical correlation](https://mlsanity.com/paper/1806.05759/) (2018)
- [Understanding image representations by measuring their equivariance and equivalence](https://mlsanity.com/paper/1411.5908/) (2014)
- [WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia](https://mlsanity.com/paper/1907.05791/) (2019)
- [Convergent Learning: Do different neural networks learn the same representations?](https://mlsanity.com/paper/1511.07543/) (2015)
- [Revisiting Model Stitching to Compare Neural Representations](https://mlsanity.com/paper/2106.07682/) (2021)
- [Exploiting Similarities among Languages for Machine Translation](https://mlsanity.com/paper/1309.4168/) (2013)
- [The Multilingual Amazon Reviews Corpus](https://mlsanity.com/paper/2010.02573/) (2020)
- [Self-Attention Between Datapoints: Going Beyond Individual Input-Output Pairs in Deep Learning](https://mlsanity.com/paper/2106.02584/) (2021)
- [Similarity and Matching of Neural Network Representations](https://mlsanity.com/paper/2110.14633/) (2021)
- [Representation Topology Divergence: A Method for Comparing Neural Network Representations](https://mlsanity.com/paper/2201.00058/) (2021)
- [Towards Understanding Learning Representations: To What Extent Do Different Neural Networks Learn the Same Representation](https://mlsanity.com/paper/1810.11750/) (2018)
- [Rethinking Channel Dimensions for Efficient Model Design](https://mlsanity.com/paper/2007.00992/) (2020)
- [The Shape of Data: Intrinsic Distance for Data Distributions](https://mlsanity.com/paper/1905.11141/) (2019)
- [Are All Good Word Vector Spaces Isomorphic?](https://mlsanity.com/paper/2004.04070/) (2020)

## Built on by (most cited first, 30 of 191)

- [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://mlsanity.com/paper/2311.05232/) (2023)
- [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](https://mlsanity.com/paper/2306.03341/) (2023)
- [The Platonic Representation Hypothesis](https://mlsanity.com/paper/2405.07987/) (2024)
- [The Geometry of Categorical and Hierarchical Concepts in Large Language Models](https://mlsanity.com/paper/2406.01506/) (2024)
- [Getting aligned on representational alignment](https://mlsanity.com/paper/2310.13018/) (2023)
- [Similarity of Neural Network Models: A Survey of Functional and Representational Measures](https://mlsanity.com/paper/2305.06329/) (2023)
- [Automatically Interpreting Millions of Features in Large Language Models](https://mlsanity.com/paper/2410.13928/) (2024)
- [Harnessing Multiple Large Language Models: A Survey on LLM Ensemble](https://mlsanity.com/paper/2502.18036/) (2025)
- [On the Origins of Linear Representations in Large Language Models](https://mlsanity.com/paper/2403.03867/) (2024)
- [Ensemble Learning for Heterogeneous Large Language Models with Deep Parallel Collaboration](https://mlsanity.com/paper/2404.12715/) (2024)
- [Text-To-Concept (and Back) via Cross-Model Alignment](https://mlsanity.com/paper/2305.06386/) (2023)
- [Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignment](https://mlsanity.com/paper/2502.03714/) (2025)
- [Harnessing the Universal Geometry of Embeddings](https://mlsanity.com/paper/2505.12540/) (2025)
- [Latent Space Translation via Semantic Alignment](https://mlsanity.com/paper/2311.00664/) (2023)
- [From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networks](https://mlsanity.com/paper/2409.14623/) (2024)
- [Steering Away from Harm: An Adaptive Approach to Defending Vision Language Model Against Jailbreaks](https://mlsanity.com/paper/2411.16721/) (2024)
- [ASIF: Coupled Data Turns Unimodal Models to Multimodal Without Training](https://mlsanity.com/paper/2210.01738/) (2022)
- [Do Vision and Language Encoders Represent the World Similarly?](https://mlsanity.com/paper/2401.05224/) (2024)
- [Assessing and Learning Alignment of Unimodal Vision and Language Models](https://mlsanity.com/paper/2412.04616/) (2024)
- [Language Models Represent Beliefs of Self and Others](https://mlsanity.com/paper/2402.18496/) (2024)
- [Unifying Causal Representation Learning with the Invariance Principle](https://mlsanity.com/paper/2409.02772/) (2024)
- [Cross-Entropy Is All You Need To Invert the Data Generating Process](https://mlsanity.com/paper/2410.21869/) (2024)
- [Learning Interpretable Concepts: Unifying Causal Representation Learning and Foundation Models](https://mlsanity.com/paper/2402.09236/) (2024)
- [There Will Be a Scientific Theory of Deep Learning](https://mlsanity.com/paper/2604.21691/) (2026)
- [What are you sinking? A geometric approach on attention sink](https://mlsanity.com/paper/2508.02546/) (2025)
- [Alignment with human representations supports robust few-shot learning](https://mlsanity.com/paper/2301.11990/) (2023)
- [Harmonics of Learning: Universal Fourier Features Emerge in Invariant Networks](https://mlsanity.com/paper/2312.08550/) (2023)
- [From Bricks to Bridges: Product of Invariances to Enhance Latent Space Communication](https://mlsanity.com/paper/2310.01211/) (2023)
- [Emotions Where Art Thou: Understanding and Characterizing the Emotional Latent Space of Large Language Models](https://mlsanity.com/paper/2510.22042/) (2025)
- [Training Neural Networks from Scratch with Parallel Low-Rank Adapters](https://mlsanity.com/paper/2402.16828/) (2024)

## Abstract (arXiv)

Neural networks embed the geometric structure of a data manifold lying in a high-dimensional space into latent representations. Ideally, the distribution of the data points in the latent space should depend only on the task, the data, the loss, and other architecture-specific constraints. However, factors such as the random weights initialization, training hyperparameters, or other sources of randomness in the training phase may induce incoherent latent spaces that hinder any form of reuse. Nevertheless, we empirically observe that, under the same data and modeling choices, the angles between the encodings within distinct latent spaces do not change. In this work, we propose the latent similarity between each sample and a fixed set of anchors as an alternative data representation, demonstrating that it can enforce the desired invariances without any additional training. We show how neural architectures can leverage these relative representations to guarantee, in practice, invariance to latent isometries and rescalings, effectively enabling latent space communication: from zero-shot model stitching to latent space comparison between diverse settings. We extensively validate the generalization capability of our approach on different datasets, spanning various modalities (images, text, graphs), tasks (e.g., classification, reconstruction) and architectures (e.g., CNNs, GCNs, transformers).