# Representation Alignment Rests on Linear Structure

arXiv: https://arxiv.org/abs/2605.28870 · HTML: https://mlsanity.com/paper/2605.28870/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 66 references that are in the map.
- First posted on arXiv 22 May 2026.

## Builds on (most cited first, 30 of 31)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Scaling and evaluating sparse autoencoders](https://mlsanity.com/paper/2406.04093/) (2024)
- [All-but-the-Top: Simple and Effective Postprocessing for Word Representations](https://mlsanity.com/paper/1702.01417/) (2017)
- [Learning Multi-Level Features with Matryoshka Sparse Autoencoders](https://mlsanity.com/paper/2503.17547/) (2025)
- [Sparse Autoencoders Do Not Find Canonical Units of Analysis](https://mlsanity.com/paper/2502.04878/) (2025)
- [On the Origins of Linear Representations in Large Language Models](https://mlsanity.com/paper/2403.03867/) (2024)
- [Revisiting the Platonic Representation Hypothesis: An Aristotelian View](https://mlsanity.com/paper/2602.14486/) (2026)
- [Steering CLIP's vision transformer with sparse autoencoders](https://mlsanity.com/paper/2504.08729/) (2025)
- [Into the Rabbit Hull: From Task-Relevant Concepts in DINO to Minkowski Geometry](https://mlsanity.com/paper/2510.08638/) (2025)
- [How Visual Representations Map to Language Feature Space in Multimodal LLMs](https://mlsanity.com/paper/2506.11976/) (2025)
- [Objective drives the consistency of representational similarity across datasets](https://mlsanity.com/paper/2411.05561/) (2024)
- [Rethinking Evaluation of Sparse Autoencoders through the Representation of Polysemous Words](https://mlsanity.com/paper/2501.06254/) (2025)
- [With Limited Data for Multimodal Alignment, Let the STRUCTURE Guide You](https://mlsanity.com/paper/2506.16895/) (2025)
- [On the Emergence of Linear Analogies in Word Embeddings](https://mlsanity.com/paper/2505.18651/) (2025)
- [Disentangling the Factors of Convergence between Brains and Computer Vision Models](https://mlsanity.com/paper/2508.18226/) (2025)
- [How Many Features Can a Language Model Store Under the Linear Representation Hypothesis?](https://mlsanity.com/paper/2602.11246/) (2026)
- [Proof of a perfect platonic representation hypothesis](https://mlsanity.com/paper/2507.01098/) (2025)
- [Global Minimizers of Sigmoid Contrastive Loss](https://mlsanity.com/paper/2509.18552/) (2025)
- [Brain-Language Model Alignment: Insights into the Platonic Hypothesis and Intermediate-Layer Advantage](https://mlsanity.com/paper/2510.17833/) (2025)
- [Training the Untrainable: Introducing Inductive Bias via Representational Alignment](https://mlsanity.com/paper/2410.20035/) (2024)
- [Canonicalizing Multimodal Contrastive Representation Learning](https://mlsanity.com/paper/2602.17584/) (2026)
- [Causes and Consequences of Representational Similarity in Machine Learning Models](https://mlsanity.com/paper/2505.13899/) (2025)
- [Better Together: Leveraging Unpaired Multimodal Data for Stronger Unimodal Models](https://mlsanity.com/paper/2510.08492/) (2025)
- [A Unified Theory of Sparse Dictionary Learning in Mechanistic Interpretability: Piecewise Biconvexity and Spurious Minima](https://mlsanity.com/paper/2512.05534/) (2025)
- [The Lattice Representation Hypothesis of Large Language Models](https://mlsanity.com/paper/2603.01227/) (2026)
- [Neuron Platonic Intrinsic Representation From Dynamics Using Contrastive Learning](https://mlsanity.com/paper/2502.10425/) (2025)
- [Escaping Plato's Cave: JAM for Aligning Independently Trained Vision and Language Models](https://mlsanity.com/paper/2507.01201/) (2025)
- [Towards Error Centric Intelligence I, Beyond Observational Learning](https://mlsanity.com/paper/2510.15128/) (2025)
- [Dreaming User Multimodal Representation Guided by The Platonic Representation Hypothesis for Micro-Video Recommendation](https://mlsanity.com/paper/2410.03538/) (2024)

## Built on by (most cited first, 1 of 1)

- [Laguerre Geometry for Interpreting Large Language Models](https://mlsanity.com/paper/2607.10578/) (2026)

## Abstract (arXiv)

We investigate the Platonic Representation Hypothesis (PRH) through a tripartite statistical framework of representations: signal, bias, and noise. {1) Signal:} We propose that Platonic alignment arises from the universal relationship between objects and attributes, which is encoded linearly in representations according to the Linear Representation Hypothesis (LRH). We provide evidence that LRH helps explain PRH by extracting linear object-attribute features with sparse autoencoders and showing that these sparse representations often exhibit stronger cross-modal alignment than their dense counterparts. {2) Bias:} Models have different implicit biases due to the diverse architectures and training procedures used. We show that this difference can be partially mitigated. Centering and normalization consistently improve cross-model alignment. {3) Noise:} Finite-sample training leads to noise in representations. We provide evidence that representational noise is driven by data scarcity by revealing a strong and consistent positive correlation between word frequency and alignment in LLMs and text embedding models. Synthesizing signal, bias, and noise, we propose a statistical model that refines the Linear Representation Hypothesis and explains further phenomena related to the alignment of representations emerging from diverse modern AI architectures.