mlsanityData to 14 Oct 2026

The Geometry of Concepts: Sparse Autoencoder Feature Structure

arXiv 2410.19750 ↗First posted 10 Oct 2024q-bio.NC cs.AI cs.LGcited by 52

On the map: Mechanistic interpretability → Sparse autoencoder training and evaluation

Abstract (arXiv)

Sparse autoencoders have recently produced dictionaries of high-dimensional vectors corresponding to the universe of concepts represented by large language models. We find that this concept universe has interesting structure at three levels: 1) The "atomic" small-scale structure contains "crystals" whose faces are parallelograms or trapezoids, generalizing well-known examples such as (man-woman-king-queen). We find that the quality of such parallelograms and associated function vectors improves greatly when projecting out global distractor directions such as word length, which is efficiently done with linear discriminant analysis. 2) The "brain" intermediate-scale structure has significant spatial modularity; for example, math and code features form a "lobe" akin to functional lobes seen in neural fMRI images. We quantify the spatial locality of these lobes with multiple metrics and find that clusters of co-occurring features, at coarse enough scale, also cluster together spatially far more than one would expect if feature geometry were random. 3) The "galaxy" scale large-scale structure of the feature point cloud is not isotropic, but instead has a power law of eigenvalues with steepest slope in middle layers. We also quantify how the clustering entropy depends on the layer.

Key facts. Cited by 52 papers in the map (data through 14 Oct 2026). Placed in Sparse autoencoder training and evaluation, one of 13 directions in Mechanistic interpretability. Builds on 28 of its 70 references that are in the map. First posted on arXiv 10 Oct 2024.

Builds on

28 of 70 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 29 Aug
  2. 2022Toy Models of Superposition21 Sep

Other references · 26

  1. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  2. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  3. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  4. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
  5. 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
  6. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 724
  7. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 716
  8. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 712
  9. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 623
  10. 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 431
  11. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 422
  12. 2023Emergent Linear Representations in World Models of Self-Supervised Sequence Modelscited by 319
14 more
  1. 2023Language Models Represent Space and Timecited by 316
  2. 2023In-Context Learning Creates Task Vectorscited by 312
  3. 2023Function Vectors in Large Language Modelscited by 308
  4. 2024Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoderscited by 300
  5. 2019Intrinsic dimension of data representations in deep neural networkscited by 282
  6. 2022The Alignment Problem from a Deep Learning Perspectivecited by 258
  7. 2023AI Deception: A Survey of Examples, Risks, and Potential Solutionscited by 188
  8. 2024The Geometry of Categorical and Hierarchical Concepts in Large Language Modelscited by 159
  9. 2021Transformer visualization via dictionary learning: contextualized embedding as a linear superposition of transformer factorscited by 108
  10. 2020Evolution of Semantic Similarity -- A Surveycited by 83
  11. 2015Sparse Overcomplete Word Vector Representationscited by 44
  12. 2024Explorations of Self-Repair in Language Modelscited by 39
  13. 2024Monotonic Representation of Numeric Properties in Language Modelscited by 15
  14. 2019Word Embedding Visualization Via Dictionary Learningcited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 22; 2026: 28.

52 papers.

Sort:

By area:

  1. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 166
  2. 2025Are Sparse Autoencoders Useful? A Case Study in Sparse Probingcited by 115
  3. 2025The Origins of Representation Manifolds in Large Language Modelscited by 51
  4. 2026When Models Manipulate Manifolds: The Geometry of a Counting Taskcited by 46
  5. 2025From Tokens to Thoughts: How LLMs and Humans Trade Compression for Meaningcited by 25
  6. 2025Beyond Prompt Engineering: Robust Behavior Control in LLMs via Steering Target Atomscited by 24
  7. 2025The Geometry of Reasoning: Flowing Logics in Representation Spacecited by 21
  8. 2026Large Language Model Reasoning Failurescited by 21
  9. 2025Rethinking Evaluation of Sparse Autoencoders through the Representation of Polysemous Wordscited by 18
  10. 2024Tracking the Feature Dynamics in LLM Training: A Mechanistic Studycited by 16
  11. 2026Do Sparse Autoencoders Identify Reasoning Features in Language Models?cited by 12
  12. 2026From Data Statistics to Feature Geometry: How Correlations Shape Superpositioncited by 10
38 more
  1. 2026Understanding Emergent Misalignment via Feature Superposition Geometrycited by 7
  2. 2025Sparse Feature Coactivation Reveals Causal Semantic Modules in Large Language Modelscited by 6
  3. 2026CSRv2: Unlocking Ultra-Sparse Embeddingscited by 6
  4. 2026From Atoms to Trees: Building a Structured Feature Forest with Hierarchical Sparse Autoencoderscited by 6
  5. 2025Analyzing Hierarchical Structure in Vision Models with Sparse Autoencoderscited by 5
  6. 2025The Shape of Adversarial Influence: Characterizing LLM Latent Spaces with Persistent Homologycited by 5
  7. 2025Harmonic Loss Trains Interpretable AI Modelscited by 4
  8. 2025On the creation of narrow AI: hierarchy and nonlocality of neural network skillscited by 4
  9. 2025Embryology of a Language Modelcited by 4
  10. 2025The Compositional Architecture of Regret in Large Language Modelscited by 3
  11. 2025Latent Concept Disentanglement in Transformer-based Language Modelscited by 3
  12. 2025Representation Learning on a Random Latticecited by 2
  13. 2026Sparse Autoencoders Map Brain-LLM Alignment onto Cortical Semantic Topographycited by 2
  14. 2025The Logical Implication Steering Method for Conditional Interventions on Transformer Generationcited by 1
  15. 2025What's in a prompt? Language models encode literary style in prompt embeddingscited by 1
  16. 2025Fusion Steering: Prompt-Specific Activation Controlcited by 1
  17. 2025Graph-Regularized Sparse Autoencoders for LLM Safety Steeringcited by 1
  18. 2026How Do Large Language Models Learn Concepts During Continual Pre-Training?cited by 1
  19. 2026Singular Vectors of Attention Heads Align with Featurescited by 1
  20. 2026The Geometric Wall: Manifold Structure Predicts Layerwise Sparse Autoencoder Scaling Lawscited by 1
  21. 2026Ablating Archetypes: The Stability of Archetypal SAEs is an Artifact of Initialization and Metric Designcited by 1
  22. 2026SAE Interventions are Unreliable: Post-Intervention Recovery of Suppressed Behaviorcited by 1
  23. 2026Do Sparse Autoencoders Learn Meaningful Concept Hierarchies?cited by 1
  24. 2026Are Single-Token Sparse Autoencoder Features Causally Necessary? Layer-Depth and SAE-Family Effectscited by 1
  25. 2026Sparse Autoencoders Encode Both Concepts and Functions: The Downstream Geometry of Feature Effectscited by 1
  26. 2026Capsule Lens: Locating and Tracking Concept Geometry in Model Representationscited by 1
  27. 2025Empirical Evaluation of Progressive Coding for Sparse Autoencoderscited by 0
  28. 2025Sparsification and Reconstruction from the Perspective of Representation Geometrycited by 0
  29. 2025Probing Geometry of Next Token Prediction Using Cumulant Expansion of the Softmax Entropycited by 0
  30. 2026Dictionary-Aligned Concept Control for Safeguarding Multimodal LLMscited by 0
  31. 2026Towards Understanding the Robustness of Sparse Autoencoderscited by 0
  32. 2026Domain-Filtered Knowledge Graphs from Sparse Autoencoder Featurescited by 0
  33. 2026From Token Lists to Graph Motifs: Weisfeiler-Lehman Analysis of Sparse Autoencoder Featurescited by 0
  34. 2026MolWorld: Molecule World Models for Actionable Molecular Optimizationcited by 0
  35. 2026Exemplar Partitioning for Mechanistic Interpretabilitycited by 0
  36. 2026Laguerre Geometry for Interpreting Large Language Modelscited by 0
  37. 2026A Survey on the Linear Representation Hypothesiscited by 0
  38. 2026Interference Beyond Geometry in Concept Extractioncited by 0

50 most cited of 52 All 52 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Do Sparse Autoencoders Capture Concept Manifolds?
  2. 2025Projecting Assumptions: The Duality Between Sparse Autoencoders and Concept Geometry
  3. 2026A Geometric View for Understanding Concept Learning and Neuron Interpretation in Sparse Autoencoders
  4. 2026Structural Instability of Feature Composition
  5. 2024Quantifying Feature Space Universality Across Large Language Models via Sparse Autoencoders
  6. 2025Understanding sparse autoencoder scaling in the presence of feature manifolds
  7. 2026Training-Driven Representational Geometry Modularization Predicts Brain Alignment in Language Models
  8. 2025A Geometric Unification of Concept Learning with Concept Cones