mlsanityData to 14 Oct 2026

Incorporating Hierarchical Semantics in Sparse Autoencoder Architectures

arXiv 2506.01197 ↗First posted 1 Jun 2025cs.CL cs.AI cs.LGcited by 14

On the map: Mechanistic interpretability → SAE features and their applications

Abstract (arXiv)

Sparse dictionary learning (and, in particular, sparse autoencoders) attempts to learn a set of human-understandable concepts that can explain variation on an abstract space. A basic limitation of this approach is that it neither exploits nor represents the semantic relationships between the learned concepts. In this paper, we introduce a modified SAE architecture that explicitly models a semantic hierarchy of concepts. Application of this architecture to the internal representations of large language models shows both that semantic hierarchy can be learned, and that doing so improves both reconstruction and interpretability. Additionally, the architecture leads to significant improvements in computational efficiency.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in SAE features and their applications, one of 13 directions in Mechanistic interpretability. Builds on 26 of its 35 references that are in the map. First posted on arXiv 1 Jun 2025.

Builds on

26 of 35 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Unveiling Language-Specific Features in Large Language Models via Sparse Autoencoders8 May
  2. 2025Sparse Shift Autoencoders for Identifying Concepts from Large Language Model Activations14 Feb
  3. 2024A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders22 Sep
  4. 2024Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders19 Jul
  5. 2024Scaling and evaluating sparse autoencoders6 Jun
  6. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Models15 Sep

Other references · 20

  1. 2018Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representationscited by 941
  2. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 716
  3. 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 435
  4. 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 394
  5. 2020Weakly-Supervised Disentanglement Without Compromisescited by 239
  6. 2024Not All Language Model Features Are One-Dimensionally Linearcited by 212
  7. 2021Towards Causal Representation Learningcited by 209
  8. 2024Transcoders Find Interpretable LLM Feature Circuitscited by 200
  9. 2024The Geometry of Categorical and Hierarchical Concepts in Large Language Modelscited by 159
  10. 2025Learning Multi-Level Features with Matryoshka Sparse Autoencoderscited by 143
  11. 2022CITRIS: Causal Identifiability from Temporal Intervened Sequencescited by 133
  12. 2024Steering Language Model Refusal with Sparse Autoencoderscited by 95
8 more
  1. 2022Weakly Supervised Representation Learning with Sparse Perturbationscited by 65
  2. 2025Projecting Assumptions: The Duality Between Sparse Autoencoders and Concept Geometrycited by 61
  3. 2024Efficient Dictionary Learning with Switch Sparse Autoencoderscited by 39
  4. 2023BISCUIT: Causal Representation Learning from Binary Interactionscited by 38
  5. 2025Automated Interpretability Metrics Do Not Distinguish Trained and Random Transformerscited by 30
  6. 2024Learning Interpretable Concepts: Unifying Causal Representation Learning and Foundation Modelscited by 29
  7. 2025Towards Interpretable Deep Generative Models via Causal Representation Learningcited by 13
  8. 2024Penzai + Treescope: A Toolkit for Interpreting, Visualizing, and Editing Models As Datacited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 12.

14 papers.

Sort:

By area:

  1. 2025Understanding sparse autoencoder scaling in the presence of feature manifoldscited by 11
  2. 2026From Reasoning Traces to Reusable Modules: Understanding Compositional Generalization in Language Model Reasoningcited by 5
  3. 2025Sparse Autoencoders are Topic Modelscited by 3
  4. 2026Tensor Product Representation Probes Reveal Shared Structure Across Linear Directionscited by 1
  5. 2026A Geometric View for Understanding Concept Learning and Neuron Interpretation in Sparse Autoencoderscited by 1
  6. 2026Do Sparse Autoencoders Learn Meaningful Concept Hierarchies?cited by 1
  7. 2026Digging Deeper: Learning Multi-Level Concept Hierarchiescited by 0
  8. 2026Dictionary-Aligned Concept Control for Safeguarding Multimodal LLMscited by 0
  9. 2026HH-SAE: Discovering and Steering Hierarchical Knowledge of Complex Manifoldscited by 0
  10. 2026Laguerre Geometry for Interpreting Large Language Modelscited by 0
  11. 2026A Dominant Diffuse Phase in the Sparse Autoencoder Phase Diagramcited by 0
  12. 2026When Trees Are Not Enough: Learning Mixed-Topology Feature Graphs with Adaptive Graph Sparse Autoencoderscited by 0
2 more
  1. 2026A Testable Theory of Atomic Featurescited by 0
  2. 2026Sparse Autoencoders for Mechanistic Interpretability in NLP: A Critical Survey of Recoverability, Interpretability, and Utilitycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026From Atoms to Trees: Building a Structured Feature Forest with Hierarchical Sparse Autoencoders
  2. 2025Temporal Sparse Autoencoders: Leveraging the Sequential Nature of Language for Interpretability
  3. 2024Disentangling Dense Embeddings with Sparse Autoencoders
  4. 2025A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Models
  5. 2025AbsTopK: Rethinking Sparse Autoencoders For Bidirectional Features
  6. 2026Tree SAE: Learning Hierarchical Feature Structures in Sparse Autoencoders
  7. 2025Beyond Redundancy: Diverse and Specialized Multi-Expert Sparse Autoencoder
  8. 2025Empirical Evaluation of Progressive Coding for Sparse Autoencoders