mlsanityData to 7 Oct 2026

A Unifying Framework for Unsupervised Concept Extraction

arXiv 2604.24936 ↗First posted 27 Apr 2026cs.LG stat.MLcited by 1

Not on the map. Why?

Abstract (arXiv)

Techniques for concept extraction, such as sparse autoencoders and transcoders, aim to extract high-level symbolic concepts from low-level nonsymbolic representations. When these extracted concepts are used for downstream tasks such as model steering and unlearning, it is essential to understand their guarantees, or lack thereof. In this work, we present a unified theoretical framework for unsupervised concept extraction, in which we frame the task of concept extraction as identifying a generative model. We present a general meta-theorem for identifiability, which reduces the problem of establishing identifiability guarantees to the problem of characterizing the intersection of two sets. As we demonstrate on a range of widely-used approaches, this meta-theorem substantially simplifies the task of proving such guarantees, thus paving the way for the development of new, principled approaches for concept extraction.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 36 of its 59 references that are in the map. First posted on arXiv 27 Apr 2026.

Builds on

36 of 59 references.

References · 36

  1. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  2. 2019An Introduction to Variational Autoencoderscited by 997
  3. 2018Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representationscited by 929
  4. 2016Deep Variational Information Bottleneckcited by 915
  5. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 679
  6. 2024Scaling and evaluating sparse autoencoderscited by 530
  7. 2016f-GAN: Training Generative Neural Samplers using Variational Divergence Minimizationcited by 504
  8. 2019The frontier of simulation-based inferencecited by 490
  9. 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 381
  10. 2024Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoderscited by 283
  11. 2018Nonlinear ICA Using Auxiliary Variables and Generalized Contrastive Learningcited by 278
  12. 2017VAE with a VampPriorcited by 247
24 more
  1. 2013k-Sparse Autoencoderscited by 209
  2. 2017Flexible statistical inference for mechanistic models of neural dynamicscited by 162
  3. 2023Getting aligned on representational alignmentcited by 142
  4. 2019From Variational to Deterministic Autoencoderscited by 131
  5. 2020On Linear Identifiability of Learned Representationscited by 103
  6. 2023Learning Linear Causal Representations from Interventions under General Nonlinear Mixingcited by 92
  7. 2025Sparse Autoencoders Trained on the Same Data Learn Different Featurescited by 86
  8. 2023Identifiability Guarantees for Causal Disentanglement from Soft Interventionscited by 84
  9. 2025Sparse Autoencoders Do Not Find Canonical Units of Analysiscited by 80
  10. 2024On the Origins of Linear Representations in Large Language Modelscited by 76
  11. 2025Projecting Assumptions: The Duality Between Sparse Autoencoders and Concept Geometrycited by 58
  12. 2019A synthetic approach to Markov kernels, conditional independence and theorems on sufficient statisticscited by 55
  13. 2025From Flat to Hierarchical: Extracting Sparse Representations with Matching Pursuitcited by 45
  14. 2022Indeterminacy in Generative Models: Characterization and Strong Identifiabilitycited by 33
  15. 2025Everything, Everywhere, All at Once: Is Mechanistic Interpretability Identifiable?cited by 32
  16. 2016Inference Compilation and Universal Probabilistic Programmingcited by 31
  17. 2024Cross-Entropy Is All You Need To Invert the Data Generating Processcited by 30
  18. 2025Simulation-Based Inference: A Practical Guidecited by 30
  19. 2013New Algorithms for Learning Incoherent and Overcomplete Dictionariescited by 22
  20. 2019Semi-Supervised Learning, Causality and the Conditional Cluster Assumptioncited by 15
  21. 2021Stable and Interpretable Unrolled Dictionary Learningcited by 15
  22. 2025Position: Mechanistic Interpretability Should Prioritize Feature Consistency in SAEscited by 14
  23. 2025CRISP: Persistent Concept Unlearning via Sparse Autoencoderscited by 9
  24. 2023SC-VAE: Sparse Coding-based Variational Autoencoder with Learned ISTAcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Concept Modulation Models: A Unified Framework for Identifiability and Extrapolationcited by 0