mlsanityData to 14 Oct 2026

The Deleuzian Representation Hypothesis

arXiv 2512.19734 ↗First posted 17 Dec 2025cs.LGcited by 0

On the map: Mechanistic interpretability → Sparse autoencoder training and evaluation

Abstract (arXiv)

We propose an alternative to sparse autoencoders (SAEs) as a simple and effective unsupervised method for extracting interpretable concepts from neural networks. The core idea is to cluster differences in activations, which we formally justify within a discriminant analysis framework. To enhance the diversity of extracted concepts, we refine the approach by weighting the clustering using the skewness of activations. The method aligns with Deleuze's modern view of concepts as differences. We evaluate the approach across five models and three modalities (vision, language, and audio), measuring concept quality, diversity, and consistency. Our results show that the proposed method achieves concept quality surpassing prior unsupervised SAE variants while approaching supervised baselines, and that the extracted concepts enable steering of a model's inner representations, demonstrating their causal influence on downstream behavior.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Sparse autoencoder training and evaluation, one of 13 directions in Mechanistic interpretability. Builds on 36 of its 65 references that are in the map. First posted on arXiv 17 Dec 2025.

Builds on

36 of 65 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Learning Multi-Level Features with Matryoshka Sparse Autoencoders21 Mar
  2. 2025Projecting Assumptions: The Duality Between Sparse Autoencoders and Concept Geometry3 Mar
  3. 2025Archetypal SAE: Adaptive and Stable Dictionary Learning for Concept Extraction in Large Vision Models18 Feb
  4. 2024Discover-then-Name: Task-Agnostic Concept Bottlenecks via Automated Concept Discovery19 Jul

Other references · 32

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  4. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  6. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  7. 2025Gemma 3 Technical Reportcited by 1,824
  8. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  9. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
  10. 2016Understanding intermediate layers using linear classifier probescited by 1,353
  11. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,319
  12. 2020Concept Bottleneck Modelscited by 1,032
20 more
  1. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 876
  2. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 767
  3. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 724
  4. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 712
  5. 2021AST: Audio Spectrogram Transformercited by 643
  6. 2024Scaling and evaluating sparse autoencoderscited by 553
  7. 2025Layer by Layer: Uncovering Hidden Representations in Language Modelscited by 311
  8. 2024Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoderscited by 300
  9. 2024Transcoders Find Interpretable LLM Feature Circuitscited by 200
  10. 2025Open Problems in Mechanistic Interpretabilitycited by 179
  11. 2024BatchTopK Sparse Autoencoderscited by 130
  12. 2023A Holistic Approach to Unifying Automatic Concept Extraction and Concept Importance Estimationcited by 109
  13. 2025Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignmentcited by 62
  14. 2025Interpreting CLIP with Hierarchical Sparse Autoencoderscited by 60
  15. 2024Towards Universality: Studying Mechanistic Similarity Across Language Model Architecturescited by 31
  16. 2025Prisma: An Open Source Toolkit for Mechanistic Interpretability in Vision and Videocited by 21
  17. 2018Unsupervised Learning of Artistic Styles with Archetypal Style Analysiscited by 10
  18. 2020Explaining Deep Neural Networks using Unsupervised Clusteringcited by 9
  19. 2022Evaluation Beyond Task Performance: Analyzing Concepts in AlphaZero in Hexcited by 8
  20. 2019CS Sparse K-means: An Algorithm for Cluster-Specific Feature Selection in High-Dimensional Clusteringcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026A Geometric View for Understanding Concept Learning and Neuron Interpretation in Sparse Autoencoders
  2. 2025Sparse Shift Autoencoders for Identifying Concepts from Large Language Model Activations
  3. 2026Concept Component Analysis: A Principled Approach for Concept Extraction in LLMs
  4. 2025Evaluating Sparse Autoencoders for Monosemantic Representation
  5. 2025Position: Use Sparse Autoencoders to Discover Unknowns
  6. 2025Interpretable Embeddings with Sparse Autoencoders: A Data Analysis Toolkit
  7. 2024Disentangling Dense Embeddings with Sparse Autoencoders
  8. 2026Do Sparse Autoencoders Learn Meaningful Concept Hierarchies?