mlsanityData to 14 Oct 2026

A Testable Theory of Atomic Features

arXiv 2610.05794 ↗First posted 5 Oct 2026cs.AI cs.LGcited by 0

On the map: Mechanistic interpretability → SAE features and their applications

Abstract (arXiv)

We develop and test a theory of language model representations in which there exist atomic features. Our main theoretical insight is that in such a model, sparse dictionaries (e.g., SAEs) of increasing size recover an increasing prefix of the most prevalent atoms in the training data. This "recovery principle" yields three testable predictions: many features in small SAEs are shared by all larger SAEs, SAEs trained on different data share features prevalent in both, and sufficiently large SAEs recover both parent and child features. In contrast to conventional wisdom that SAE features are unstable and "split" as size increases, we find that these predictions hold on SAEs of sizes ranging from 512 to 131,072 trained on two large embedding models. From a theoretical perspective, our results suggest the promise of a scientific theory of representations based on atomic features. Practically, our results suggest the promise of scaling SAEs.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in SAE features and their applications, one of 13 directions in Mechanistic interpretability. Builds on 35 of its 77 references that are in the map. First posted on arXiv 5 Oct 2026.

Builds on

35 of 77 references.

Closest ancestors · 6

Same direction, newest first

  1. 2026Tree SAE: Learning Hierarchical Feature Structures in Sparse Autoencoders8 May
  2. 2025Incorporating Hierarchical Semantics in Sparse Autoencoder Architectures1 Jun
  3. 2025Sparse Autoencoders for Hypothesis Generation5 Feb
  4. 2025Sparse Autoencoders Trained on the Same Data Learn Different Features28 Jan
  5. 2024Scaling and evaluating sparse autoencoders6 Jun
  6. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Models15 Sep

Other references · 29

  1. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  2. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
  3. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  4. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
  5. 2022Toy Models of Superpositioncited by 971
  6. 2023Steering Language Models With Activation Engineeringcited by 922
  7. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 716
  8. 2024The Platonic Representation Hypothesiscited by 382
  9. 2025Persona Vectors: Monitoring and Controlling Character Traits in Language Modelscited by 345
  10. 2023Emergent Linear Representations in World Models of Self-Supervised Sequence Modelscited by 319
  11. 2013k-Sparse Autoencoderscited by 215
  12. 2025Learning Multi-Level Features with Matryoshka Sparse Autoencoderscited by 143
17 more
  1. 2025Persona Features Control Emergent Misalignmentcited by 86
  2. 2025Archetypal SAE: Adaptive and Stable Dictionary Learning for Concept Extraction in Large Vision Modelscited by 73
  3. 2025Interpreting CLIP with Hierarchical Sparse Autoencoderscited by 60
  4. 2025Llama-Embed-Nemotron-8B: A Universal Text Embedding Model for Multilingual and Cross-Lingual Taskscited by 51
  5. 2025From Flat to Hierarchical: Extracting Sparse Representations with Matching Pursuitcited by 49
  6. 2025(How) Do Language Models Track State?cited by 34
  7. 2025Into the Rabbit Hull: From Task-Relevant Concepts in DINO to Minkowski Geometrycited by 34
  8. 2026Do Sparse Autoencoders Capture Concept Manifolds?cited by 31
  9. 2025Position: Use Sparse Autoencoders to Discover Unknownscited by 19
  10. 2025Sparse but Wrong: Incorrect L0 Leads to Incorrect Features in Sparse Autoencoderscited by 14
  11. 2024Competition and Diversity in Generative AIcited by 13
  12. 2026Gemini Embedding 2: A Native Multimodal Embedding Model from Geminicited by 10
  13. 2026EconCSLib: AI-Assisted Lean Formalization for Economics & Computation researchcited by 7
  14. 2026Structuring Sparsity: Block-Sparse Featurizers Capture Visual Concept Manifoldscited by 5
  15. 2026The Subjectivity of Monoculturecited by 2
  16. 2026Characterizing Universal Object Representations Across Vision Modelscited by 2
  17. 2014An Incidence Geometry approach to Dictionary Learningcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Towards Atoms of Large Language Models
  2. 2026Descriptive Collision in Sparse Autoencoder Auto-Interpretability: When One Explanation Describes Many Features
  3. 2026Sparse Autoencoders for Mechanistic Interpretability in NLP: A Critical Survey of Recoverability, Interpretability, and Utility
  4. 2026How Pruning Reshapes Features: Sparse Autoencoder Analysis of Weight-Pruned Language Models
  5. 2025A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Models
  6. 2026From Atoms to Trees: Building a Structured Feature Forest with Hierarchical Sparse Autoencoders
  7. 2026Qwen-Scope: Turning Sparse Features into Development Tools for Large Language Models
  8. 2024A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders