# Do Sparse Autoencoders Learn Meaningful Concept Hierarchies?

arXiv: https://arxiv.org/abs/2606.22994 · HTML: https://mlsanity.com/paper/2606.22994/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Sparse autoencoder training and evaluation, one of 13 directions in Mechanistic interpretability.
- Builds on 38 of its 59 references that are in the map.
- First posted on arXiv 22 Jun 2026.

## Placed in

- Mechanistic interpretability / [Sparse autoencoder training and evaluation](https://mlsanity.com/mechinterp/sparse-autoencoder-training-and-evaluation/)

## Builds on (most cited first, 30 of 38)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017)
- [Concept Bottleneck Models](https://mlsanity.com/paper/2007.04612/) (2020)
- [Manipulating and Measuring Model Interpretability](https://mlsanity.com/paper/1802.07810/) (2018)
- [A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders](https://mlsanity.com/paper/2409.14507/) (2024)
- [The Geometry of Categorical and Hierarchical Concepts in Large Language Models](https://mlsanity.com/paper/2406.01506/) (2024)
- [Learning Multi-Level Features with Matryoshka Sparse Autoencoders](https://mlsanity.com/paper/2503.17547/) (2025)
- [SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability](https://mlsanity.com/paper/2503.09532/) (2025)
- [Automatically Interpreting Millions of Features in Large Language Models](https://mlsanity.com/paper/2410.13928/) (2024)
- [A Holistic Approach to Unifying Automatic Concept Extraction and Concept Importance Estimation](https://mlsanity.com/paper/2306.07304/) (2023)
- [Concept-based Explainable Artificial Intelligence: A Survey](https://mlsanity.com/paper/2312.12936/) (2023)
- [Discover-then-Name: Task-Agnostic Concept Bottlenecks via Automated Concept Discovery](https://mlsanity.com/paper/2407.14499/) (2024)
- [Sparse Autoencoders Do Not Find Canonical Units of Analysis](https://mlsanity.com/paper/2502.04878/) (2025)
- [SAEs Are Good for Steering -- If You Select the Right Features](https://mlsanity.com/paper/2505.20063/) (2025)
- [A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Models](https://mlsanity.com/paper/2503.05613/) (2025)
- [Sparse Autoencoders Learn Monosemantic Features in Vision-Language Models](https://mlsanity.com/paper/2504.02821/) (2025)
- [Domain-specific Knowledge Graphs: A survey](https://mlsanity.com/paper/2011.00235/) (2020)
- [A cost function for similarity-based hierarchical clustering](https://mlsanity.com/paper/1510.05043/) (2015)
- [Interpreting CLIP with Hierarchical Sparse Autoencoders](https://mlsanity.com/paper/2502.20578/) (2025)
- [Controllable Text Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2408.12599/) (2024)
- [Leveraging Explanations in Interactive Machine Learning: An Overview](https://mlsanity.com/paper/2207.14526/) (2022)
- [The Geometry of Concepts: Sparse Autoencoder Feature Structure](https://mlsanity.com/paper/2410.19750/) (2024)
- [From Flat to Hierarchical: Extracting Sparse Representations with Matching Pursuit](https://mlsanity.com/paper/2506.03093/) (2025)
- [Mechanistic understanding and validation of large AI models with SemanticLens](https://mlsanity.com/paper/2501.05398/) (2025)
- [Further Generalizations of the Jaccard Index](https://mlsanity.com/paper/2110.09619/) (2021)
- [Chain-of-Layer: Iteratively Prompting Large Language Models for Taxonomy Induction from Limited Examples](https://mlsanity.com/paper/2402.07386/) (2024)
- [Eliminating Information Leakage in Hard Concept Bottleneck Models with Supervised, Hierarchical Concept Learning](https://mlsanity.com/paper/2402.05945/) (2024)
- [Feature Hedging: Correlated Features Break Narrow Sparse Autoencoders](https://mlsanity.com/paper/2505.11756/) (2025)

## Built on by (most cited first, 1 of 1)

- [When Trees Are Not Enough: Learning Mixed-Topology Feature Graphs with Adaptive Graph Sparse Autoencoders](https://mlsanity.com/paper/2609.36294/) (2026)

## Abstract (arXiv)

Sparse autoencoders (SAEs) have become an important tool for unsupervised concept discovery in large models. To make the resulting feature spaces more interpretable and manageable, recent approaches have begun imposing hierarchical structure, either explicitly or as an implicit effect of training constraints, yet rigorous comparison remains difficult. There are no agreed-upon requirements for what a meaningful feature hierarchy should satisfy, and evaluation has largely relied on qualitative illustrations with fragmented quantitative protocols. To address this, we derive a set of key requirements for generalization/specialization hierarchies in unsupervised concept discovery, drawing on semantic net and taxonomy research alongside recent SAE work, and use them to derive a concrete evaluation protocol. Applying this protocol to current SAE approaches trained on visual data, we find that while feature spaces generally provide a basis for sensible hierarchies, establishing good hierarchical structure remains challenging. In particular, feature absorption, both in its well-known hard form and in a continuous, soft form, systematically compromises hierarchy quality, pointing to a fundamental tension that future approaches will need to navigate.