mlsanityData to 14 Oct 2026

The Origins of Representation Manifolds in Large Language Models

arXiv 2505.18235 ↗First posted 23 May 2025cs.LG cs.AIcited by 51

On the map: Mechanistic interpretability → Sparse autoencoder training and evaluation

Abstract (arXiv)

There is a large ongoing scientific effort in mechanistic interpretability to map embeddings and internal representations of AI systems into human-understandable concepts. A key element of this effort is the linear representation hypothesis, which posits that neural representations are sparse linear combinations of `almost-orthogonal' direction vectors, reflecting the presence or absence of different features. This model underpins the use of sparse autoencoders to recover features from representations. Moving towards a fuller model of features, in which neural representations could encode not just the presence but also a potentially continuous and multidimensional value for a feature, has been a subject of intense recent discourse. We describe why and how a feature might be represented as a manifold, demonstrating in particular that cosine similarity in representation space may encode the intrinsic geometry of a feature through shortest, on-manifold paths, potentially answering the question of how distance in representation space and relatedness in concept space could be connected. The critical assumptions and predictions of the theory are validated on text embeddings and token activations of large language models.

Key facts. Cited by 51 papers in the map (data through 14 Oct 2026). Placed in Sparse autoencoder training and evaluation, one of 13 directions in Mechanistic interpretability. Builds on 24 of its 49 references that are in the map. First posted on arXiv 23 May 2025.

Builds on

24 of 49 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Learning Multi-Level Features with Matryoshka Sparse Autoencoders21 Mar
  2. 2024The Geometry of Concepts: Sparse Autoencoder Feature Structure10 Oct
  3. 2022Toy Models of Superposition21 Sep

Other references · 21

  1. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  2. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
  3. 2016Understanding intermediate layers using linear classifier probescited by 1,353
  4. 2023Progress measures for grokking via mechanistic interpretabilitycited by 839
  5. 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 431
  6. 2023Emergent Linear Representations in World Models of Self-Supervised Sequence Modelscited by 319
  7. 2023Language Models Represent Space and Timecited by 316
  8. 2018Towards a Definition of Disentangled Representationscited by 296
  9. 2019Intrinsic dimension of data representations in deep neural networkscited by 282
  10. 2022Towards Understanding Grokking: An Effective Theory of Representation Learningcited by 230
  11. 2021Contrastive Learning Inverts the Data Generating Processcited by 221
  12. 2024Not All Language Model Features Are One-Dimensionally Linearcited by 212
9 more
  1. 2016Linear Algebraic Structure of Word Senses, with Applications to Polysemycited by 165
  2. 2024The Geometry of Categorical and Hierarchical Concepts in Large Language Modelscited by 159
  3. 2023The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networkscited by 156
  4. 2022The Geometry of Multilingual Language Model Representationscited by 95
  5. 2024Transformers represent belief state geometry in their residual streamcited by 77
  6. 2023Identifiability of latent-variable and structural-equation models: from linear to nonlinearcited by 58
  7. 2024Learning to grok: Emergence of in-context learning and skill composition in modular arithmetic taskscited by 40
  8. 2024Cross-Entropy Is All You Need To Invert the Data Generating Processcited by 31
  9. 2020Disentangling by Subspace Diffusioncited by 28

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 44.

51 papers.

Sort:

By area:

  1. 2026When Models Manipulate Manifolds: The Geometry of a Counting Taskcited by 46
  2. 2026Do Sparse Autoencoders Capture Concept Manifolds?cited by 31
  3. 2026Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behaviorcited by 26
  4. 2025The Geometry of Reasoning: Flowing Logics in Representation Spacecited by 21
  5. 2025Understanding sparse autoencoder scaling in the presence of feature manifoldscited by 11
  6. 2026The Information Geometry of Softmax: Probing and Steeringcited by 11
  7. 2026From Data Statistics to Feature Geometry: How Correlations Shape Superpositioncited by 10
  8. 2026Arithmetic in the Wild: Llama uses Base-10 Addition to Reason About Cyclic Conceptscited by 10
  9. 2025Hypothesis-Driven Feature Manifold Analysis in LLMs via Supervised Multi-Dimensional Scalingcited by 8
  10. 2026The Geometry of Representational Failures in Vision Language Modelscited by 6
  11. 2026Beyond Steering Vector: Flow-based Activation Steering for Inference-Time Interventioncited by 6
  12. 2025Finding Manifolds With Bilinear Autoencoderscited by 5
38 more
  1. 2026The Cylindrical Representation Hypothesis for Language Model Steeringcited by 5
  2. 2026Structuring Sparsity: Block-Sparse Featurizers Capture Visual Concept Manifoldscited by 5
  3. 2026MolEvolve: LLM-Guided Evolutionary Search for Interpretable Molecular Optimizationcited by 4
  4. 2026Rethinking Layer Relevance in Large Language Models Beyond Cosine Similaritycited by 4
  5. 2026Subspace-Aware Sparse Autoencoders for Effective Mechanistic Interpretabilitycited by 4
  6. 2026ELROND: Exploring and decomposing intrinsic capabilities of diffusion modelscited by 2
  7. 2025Latent Trajectory Dynamics in Large Language Models: A Manifold Evolution Framework with Empirical Validationcited by 1
  8. 2025Binary Autoencoder for Mechanistic Interpretability of Large Language Modelscited by 1
  9. 2026Bilinear autoencoders find interpretable manifoldscited by 1
  10. 2026SMIXAE: Towards Unsupervised Manifold Discovery in Language Modelscited by 1
  11. 2026The Geometric Wall: Manifold Structure Predicts Layerwise Sparse Autoencoder Scaling Lawscited by 1
  12. 2026Tensor Product Representation Probes Reveal Shared Structure Across Linear Directionscited by 1
  13. 2026Probing for Representation Manifolds in Superpositioncited by 1
  14. 2026Latent-space Attacks for Refusal Evasion in Language Modelscited by 1
  15. 2026Hierarchical Concept Geometry in Language Models Emerges from Word Co-occurrencecited by 1
  16. 2026Riemannian-Manifold Steering: Geometry-Aware Generative Autoencoders for Label-Free Steeringcited by 1
  17. 2026Geometry of Human Perceptual Domains Emerges Transiently in LLM Representationscited by 1
  18. 2026When and How Long? The Readout-Mediator Angle in Temporal Reasoningcited by 1
  19. 2026Capsule Lens: Locating and Tracking Concept Geometry in Model Representationscited by 1
  20. 2025MindCraft: How Concept Trees Take Shape In Deep Modelscited by 0
  21. 2026Multilingual Embedding Probes Fail to Generalize Across Learner Corporacited by 0
  22. 2026Temporal Preference Concepts and their Functions in a Large Language Modelcited by 0
  23. 2026LoRi: Low-Rank Distillation for Implicit Reasoningcited by 0
  24. 2026A Navigable Manifold of Hypothesized Consciousness-Spectrum States in Language Model Representationscited by 0
  25. 2026Manifold Bandits: Bayesian Curriculum Learning over the Latent Geometry of Large Language Modelscited by 0
  26. 2026Does Aurora Encode Atmospheric Structure? Latent Regime Analysis and Attributioncited by 0
  27. 2026At the Edge of Understanding: Sparse Autoencoders Trace The Limits of Transformer Generalizationcited by 0
  28. 2026Subliminal Clocks: Latent Time Modelling in Diffusion Language Modelscited by 0
  29. 2026Geometry of Ordinal Representations in Language Modelscited by 0
  30. 2026What, Where, and How: Probing Spatiotemporal Representations in Video Foundation Modelscited by 0
  31. 2026MAxBench: A Multinomial Concept Recovery Benchmarkcited by 0
  32. 2026Xeno-Interpretability: Investigating the Alien Minds of LLMscited by 0
  33. 2026ManiEdit: Sequential Unstructured Knowledge Editing for Language Models from a Manifold Perspectivecited by 0
  34. 2026Causal and Interpretable Structures in LLM Compositional Taskscited by 0
  35. 2026When Models Don't Manipulate Manifolds: The Geometry of a Comparison Taskcited by 0
  36. 2026MetaSteer: Context-Conditioned, nonlinear Steering via Attention-Projection Adaptationcited by 0
  37. 2026The Geometry of Contextual Relations: Language Models Address Facts by Order of Mentioncited by 0
  38. 2026Tucker Bottleneck Attention for Multi-Dimensional Sequence Modelingcited by 0

50 most cited of 51 All 51 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Quantifying Feature Space Universality Across Large Language Models via Sparse Autoencoders
  2. 2023The Linear Representation Hypothesis and the Geometry of Large Language Models
  3. 2026Beyond Linear Concepts: Discovering and Aligning Non-Linear Concept Manifolds in Large Language Models
  4. 2025Sparsification and Reconstruction from the Perspective of Representation Geometry
  5. 2025Shared Global and Local Geometry of Language Model Embeddings
  6. 2026Representational Capacity: Geometric Limits on Feature Representation in Transformer Language Models
  7. 2026Emergent Manifold Separability during Reasoning in Large Language Models
  8. 2025Towards Atoms of Large Language Models