mlsanityData to 7 Oct 2026

Not All Language Model Features Are One-Dimensionally Linear

arXiv 2405.14860 ↗First posted 23 May 2024cs.LGcited by 201

Not on the map. Why?

Abstract (arXiv)

Recent work has proposed that language models perform computation by manipulating one-dimensional representations of concepts ("features") in activation space. In contrast, we explore whether some language model representations may be inherently multi-dimensional. We begin by developing a rigorous definition of irreducible multi-dimensional features based on whether they can be decomposed into either independent or non-co-occurring lower-dimensional features. Motivated by these definitions, we design a scalable method that uses sparse autoencoders to automatically find multi-dimensional features in GPT-2 and Mistral 7B. These auto-discovered features include strikingly interpretable examples, e.g. circular features representing days of the week and months of the year. We identify tasks where these exact circles are used to solve computational problems involving modular arithmetic in days of the week and months of the year. Next, we provide evidence that these circular features are indeed the fundamental unit of computation in these tasks with intervention experiments on Mistral 7B and Llama 3 8B, and we examine the continuity of the days of the week feature in Mistral 7B. Overall, our work argues that understanding multi-dimensional features is necessary to mechanistically decompose some model behaviors.

Key facts. Cited by 201 papers in the map (data through 7 Oct 2026). Builds on 31 of its 57 references that are in the map. First posted on arXiv 23 May 2024.

Builds on

31 of 57 references.

References · 31

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  3. 2023Mistral 7Bcited by 3,065
  4. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  5. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  6. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  7. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  8. 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
  9. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
  10. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 679
  11. 2023Instruction Tuning with GPT-4cited by 676
  12. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 673
19 more
  1. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  2. 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 417
  3. 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 412
  4. 2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 328
  5. 2023Emergent Linear Representations in World Models of Self-Supervised Sequence Modelscited by 313
  6. 2023Language Models Represent Space and Timecited by 306
  7. 2022Towards Understanding Grokking: An Effective Theory of Representation Learningcited by 229
  8. 2023How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language modelcited by 199
  9. 2023Attribution Patching Outperforms Automated Circuit Discoverycited by 196
  10. 2023The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networkscited by 153
  11. 2024Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systemscited by 94
  12. 2023A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysiscited by 88
  13. 2023Successor Heads: Recurring, Interpretable Attention Heads In The Wildcited by 77
  14. 2024On the Origins of Linear Representations in Large Language Modelscited by 76
  15. 2022Interpreting Neural Networks through the Polytope Lenscited by 33
  16. 2023Feature emergence via margin maximization: case studies in algebraic taskscited by 33
  17. 2023Provably safe systems: the only path to controllable AGIcited by 29
  18. 2024Opening the AI black box: program synthesis via mechanistic interpretabilitycited by 19
  19. 2024Monotonic Representation of Numeric Properties in Language Modelscited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 21; 2025: 61; 2026: 119.

201 papers.

Sort:

By area:

  1. 2024A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoderscited by 148
  2. 2024Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoderscited by 133
  3. 2024KAN 2.0: Kolmogorov-Arnold Networks Meet Sciencecited by 116
  4. 2025Are Sparse Autoencoders Useful? A Case Study in Sparse Probingcited by 109
  5. 2024Transformers represent belief state geometry in their residual streamcited by 73
  6. 2025The Origins of Representation Manifolds in Large Language Modelscited by 46
  7. 2025From Flat to Hierarchical: Extracting Sparse Representations with Matching Pursuitcited by 45
  8. 2026When Models Manipulate Manifolds: The Geometry of a Counting Taskcited by 43
  9. 2024Evaluating Open-Source Sparse Autoencoders on Disentangling Factual Knowledge in GPT-2 Smallcited by 34
  10. 2025The Non-Linear Representation Dilemma: Is Causal Abstraction Enough for Mechanistic Interpretability?cited by 34
  11. 2024Language Models Encode Numbers Using Digit Representations in Base 10cited by 32
  12. 2025Probing and Steering Evaluation Awareness of Language Modelscited by 31
38 more
  1. 2024Cross-Entropy Is All You Need To Invert the Data Generating Processcited by 30
  2. 2024Representational Analysis of Binding in Language Modelscited by 27
  3. 2025Towards Understanding Distilled Reasoning Models: A Representational Approachcited by 26
  4. 2024Quantifying Feature Space Universality Across Large Language Models via Sparse Autoencoderscited by 25
  5. 2026Do Sparse Autoencoders Capture Concept Manifolds?cited by 25
  6. 2024InterpBench: Semi-Synthetic Transformers for Evaluating Mechanistic Interpretability Techniquescited by 24
  7. 2024The Quest for the Right Mediator: Surveying Mechanistic Interpretability Through the Lens of Causal Mediation Analysiscited by 24
  8. 2024Relational Composition in Neural Networks: A Survey and Call to Actioncited by 23
  9. 2026Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behaviorcited by 22
  10. 2025The Geometry of Reasoning: Flowing Logics in Representation Spacecited by 21
  11. 2026There Will Be a Scientific Theory of Deep Learningcited by 21
  12. 2024RoseLoRA: Row and Column-wise Sparse Low-rank Adaptation of Pre-trained Language Model for Knowledge Editing and Fine-tuningcited by 20
  13. 2024Disentangling Dense Embeddings with Sparse Autoencoderscited by 19
  14. 2024Residual Stream Analysis with Multi-Layer SAEscited by 18
  15. 2024Robust AI-Generated Text Detection by Restricted Embeddingscited by 18
  16. 2026Deriving Neural Scaling Laws from the statistics of natural languagecited by 18
  17. 2025Transferring Linear Features Across Language Models With Model Stitchingcited by 17
  18. 2025Probing for Arithmetic Errors in Language Modelscited by 17
  19. 2025Feature Hedging: Correlated Features Break Narrow Sparse Autoencoderscited by 15
  20. 2024Geometric Signatures of Compositionality Across a Language Model's Lifetimecited by 14
  21. 2025Constrained belief updates explain geometric structures in transformer representationscited by 14
  22. 2025Large Language Models Encode Semantics and Alignment in Linearly Separable Representationscited by 14
  23. 2025Beyond Linear Probes: Dynamic Safety Monitoring for Language Modelscited by 14
  24. 2025Analyze Feature Flow to Enhance Interpretation and Steering in Language Modelscited by 12
  25. 2025Incorporating Hierarchical Semantics in Sparse Autoencoder Architecturescited by 12
  26. 2024Persistent Topological Features in Large Language Modelscited by 11
  27. 2025Analogical Reasoning Inside Large Language Models: Concept Vectors and the Limits of Abstractioncited by 11
  28. 2025Internal Causal Mechanisms Robustly Predict Language Model Out-of-Distribution Behaviorscited by 11
  29. 2025Sparse but Wrong: Incorrect L0 Leads to Incorrect Features in Sparse Autoencoderscited by 11
  30. 2025Understanding sparse autoencoder scaling in the presence of feature manifoldscited by 11
  31. 2026Do Sparse Autoencoders Identify Reasoning Features in Language Models?cited by 11
  32. 2026How Many Features Can a Language Model Store Under the Linear Representation Hypothesis?cited by 11
  33. 2025How Causal Abstraction Underpins Computational Explanationcited by 10
  34. 2026From Directions to Regions: Decomposing Activations in Language Models via Local Geometrycited by 10
  35. 2025SOM Directions are Better than One: Multi-Directional Refusal Suppression in Language Modelscited by 9
  36. 2026The Information Geometry of Softmax: Probing and Steeringcited by 9
  37. 2026From Data Statistics to Feature Geometry: How Correlations Shape Superpositioncited by 9
  38. 2026Tool Calling is Linearly Readable and Steerable in Language Modelscited by 9

50 most cited of 201 All 201 (JSON)