# The Origins of Representation Manifolds in Large Language Models

arXiv: https://arxiv.org/abs/2505.18235 · HTML: https://mlsanity.com/paper/2505.18235/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- Placed in Sparse autoencoder training and evaluation, one of 13 directions in Mechanistic interpretability.
- Builds on 24 of its 49 references that are in the map.
- First posted on arXiv 23 May 2025.

## Placed in

- Mechanistic interpretability / [Sparse autoencoder training and evaluation](https://mlsanity.com/mechinterp/sparse-autoencoder-training-and-evaluation/)

## Builds on (most cited first, 24 of 24)

- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [Toy Models of Superposition](https://mlsanity.com/paper/2209.10652/) (2022)
- [Progress measures for grokking via mechanistic interpretability](https://mlsanity.com/paper/2301.05217/) (2023)
- [Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task](https://mlsanity.com/paper/2210.13382/) (2022)
- [Emergent Linear Representations in World Models of Self-Supervised Sequence Models](https://mlsanity.com/paper/2309.00941/) (2023)
- [Language Models Represent Space and Time](https://mlsanity.com/paper/2310.02207/) (2023)
- [Towards a Definition of Disentangled Representations](https://mlsanity.com/paper/1812.02230/) (2018)
- [Intrinsic dimension of data representations in deep neural networks](https://mlsanity.com/paper/1905.12784/) (2019)
- [Towards Understanding Grokking: An Effective Theory of Representation Learning](https://mlsanity.com/paper/2205.10343/) (2022)
- [Contrastive Learning Inverts the Data Generating Process](https://mlsanity.com/paper/2102.08850/) (2021)
- [Not All Language Model Features Are One-Dimensionally Linear](https://mlsanity.com/paper/2405.14860/) (2024)
- [Linear Algebraic Structure of Word Senses, with Applications to Polysemy](https://mlsanity.com/paper/1601.03764/) (2016)
- [The Geometry of Categorical and Hierarchical Concepts in Large Language Models](https://mlsanity.com/paper/2406.01506/) (2024)
- [The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networks](https://mlsanity.com/paper/2306.17844/) (2023)
- [Learning Multi-Level Features with Matryoshka Sparse Autoencoders](https://mlsanity.com/paper/2503.17547/) (2025)
- [The Geometry of Multilingual Language Model Representations](https://mlsanity.com/paper/2205.10964/) (2022)
- [Transformers represent belief state geometry in their residual stream](https://mlsanity.com/paper/2405.15943/) (2024)
- [Identifiability of latent-variable and structural-equation models: from linear to nonlinear](https://mlsanity.com/paper/2302.02672/) (2023)
- [The Geometry of Concepts: Sparse Autoencoder Feature Structure](https://mlsanity.com/paper/2410.19750/) (2024)
- [Learning to grok: Emergence of in-context learning and skill composition in modular arithmetic tasks](https://mlsanity.com/paper/2406.02550/) (2024)
- [Cross-Entropy Is All You Need To Invert the Data Generating Process](https://mlsanity.com/paper/2410.21869/) (2024)
- [Disentangling by Subspace Diffusion](https://mlsanity.com/paper/2006.12982/) (2020)

## Built on by (most cited first, 30 of 51)

- [When Models Manipulate Manifolds: The Geometry of a Counting Task](https://mlsanity.com/paper/2601.04480/) (2026)
- [Do Sparse Autoencoders Capture Concept Manifolds?](https://mlsanity.com/paper/2604.28119/) (2026)
- [Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behavior](https://mlsanity.com/paper/2605.05115/) (2026)
- [The Geometry of Reasoning: Flowing Logics in Representation Space](https://mlsanity.com/paper/2510.09782/) (2025)
- [Understanding sparse autoencoder scaling in the presence of feature manifolds](https://mlsanity.com/paper/2509.02565/) (2025)
- [The Information Geometry of Softmax: Probing and Steering](https://mlsanity.com/paper/2602.15293/) (2026)
- [From Data Statistics to Feature Geometry: How Correlations Shape Superposition](https://mlsanity.com/paper/2603.09972/) (2026)
- [Arithmetic in the Wild: Llama uses Base-10 Addition to Reason About Cyclic Concepts](https://mlsanity.com/paper/2605.01148/) (2026)
- [Hypothesis-Driven Feature Manifold Analysis in LLMs via Supervised Multi-Dimensional Scaling](https://mlsanity.com/paper/2510.01025/) (2025)
- [The Geometry of Representational Failures in Vision Language Models](https://mlsanity.com/paper/2602.07025/) (2026)
- [Beyond Steering Vector: Flow-based Activation Steering for Inference-Time Intervention](https://mlsanity.com/paper/2605.05892/) (2026)
- [Finding Manifolds With Bilinear Autoencoders](https://mlsanity.com/paper/2510.16820/) (2025)
- [The Cylindrical Representation Hypothesis for Language Model Steering](https://mlsanity.com/paper/2605.01844/) (2026)
- [Structuring Sparsity: Block-Sparse Featurizers Capture Visual Concept Manifolds](https://mlsanity.com/paper/2606.25234/) (2026)
- [MolEvolve: LLM-Guided Evolutionary Search for Interpretable Molecular Optimization](https://mlsanity.com/paper/2603.24382/) (2026)
- [Rethinking Layer Relevance in Large Language Models Beyond Cosine Similarity](https://mlsanity.com/paper/2605.14075/) (2026)
- [Subspace-Aware Sparse Autoencoders for Effective Mechanistic Interpretability](https://mlsanity.com/paper/2606.06333/) (2026)
- [ELROND: Exploring and decomposing intrinsic capabilities of diffusion models](https://mlsanity.com/paper/2602.10216/) (2026)
- [Latent Trajectory Dynamics in Large Language Models: A Manifold Evolution Framework with Empirical Validation](https://mlsanity.com/paper/2505.20340/) (2025)
- [Binary Autoencoder for Mechanistic Interpretability of Large Language Models](https://mlsanity.com/paper/2509.20997/) (2025)
- [Bilinear autoencoders find interpretable manifolds](https://mlsanity.com/paper/2605.08891/) (2026)
- [SMIXAE: Towards Unsupervised Manifold Discovery in Language Models](https://mlsanity.com/paper/2605.09224/) (2026)
- [The Geometric Wall: Manifold Structure Predicts Layerwise Sparse Autoencoder Scaling Laws](https://mlsanity.com/paper/2605.09887/) (2026)
- [Tensor Product Representation Probes Reveal Shared Structure Across Linear Directions](https://mlsanity.com/paper/2605.09967/) (2026)
- [Probing for Representation Manifolds in Superposition](https://mlsanity.com/paper/2605.18537/) (2026)
- [Latent-space Attacks for Refusal Evasion in Language Models](https://mlsanity.com/paper/2605.21706/) (2026)
- [Hierarchical Concept Geometry in Language Models Emerges from Word Co-occurrence](https://mlsanity.com/paper/2605.23821/) (2026)
- [Riemannian-Manifold Steering: Geometry-Aware Generative Autoencoders for Label-Free Steering](https://mlsanity.com/paper/2605.24942/) (2026)
- [Geometry of Human Perceptual Domains Emerges Transiently in LLM Representations](https://mlsanity.com/paper/2605.27970/) (2026)
- [When and How Long? The Readout-Mediator Angle in Temporal Reasoning](https://mlsanity.com/paper/2605.29126/) (2026)

## Abstract (arXiv)

There is a large ongoing scientific effort in mechanistic interpretability to map embeddings and internal representations of AI systems into human-understandable concepts. A key element of this effort is the linear representation hypothesis, which posits that neural representations are sparse linear combinations of `almost-orthogonal' direction vectors, reflecting the presence or absence of different features. This model underpins the use of sparse autoencoders to recover features from representations. Moving towards a fuller model of features, in which neural representations could encode not just the presence but also a potentially continuous and multidimensional value for a feature, has been a subject of intense recent discourse. We describe why and how a feature might be represented as a manifold, demonstrating in particular that cosine similarity in representation space may encode the intrinsic geometry of a feature through shortest, on-manifold paths, potentially answering the question of how distance in representation space and relatedness in concept space could be connected. The critical assumptions and predictions of the theory are validated on text embeddings and token activations of large language models.