mlsanityData to 14 Oct 2026

Interpretable and Steerable Concept Bottleneck Sparse Autoencoders

arXiv 2512.10805 ↗First posted 11 Dec 2025cs.LG cs.CVcited by 6

On the map: Mechanistic interpretability → Sparse autoencoder training and evaluation

Abstract (arXiv)

Sparse autoencoders (SAEs) promise a unified approach for mechanistic interpretability, concept discovery, and model steering in LLMs and LVLMs. However, realizing this potential requires learned features to be both interpretable and steerable. To that end, we introduce two new computationally inexpensive interpretability and steerability metrics for a systematic analysis of LVLM SAEs. This uncovers two observations; (i) a majority of SAE neurons exhibit either low interpretability or low steerability or both, rendering them ineffective for downstream use; and (ii) user-desired concepts are often absent in the SAE, thus limiting their practical utility. To address these limitations, we propose Concept Bottleneck Sparse Autoencoders (CB-SAE) - a novel post-hoc framework that prunes low-utility neurons and augments the latent space with a lightweight concept bottleneck aligned to a user-defined concept set. The resulting CB-SAE improves interpretability by +32.1% and steerability by +14.5% across LVLMs and image generation tasks.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Sparse autoencoder training and evaluation, one of 13 directions in Mechanistic interpretability. Builds on 59 of its 66 references that are in the map. First posted on arXiv 11 Dec 2025.

Builds on

59 of 66 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025From Flat to Hierarchical: Extracting Sparse Representations with Matching Pursuit3 Jun
  2. 2025Learning Multi-Level Features with Matryoshka Sparse Autoencoders21 Mar
  3. 2025Archetypal SAE: Adaptive and Stable Dictionary Learning for Concept Extraction in Large Vision Models18 Feb
  4. 2025Transcoders Beat Sparse Autoencoders for Interpretability31 Jan
  5. 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 29 Aug
  6. 2024Discover-then-Name: Task-Agnostic Concept Bottlenecks via Automated Concept Discovery19 Jul

Other references · 53

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  6. 2023Visual Instruction Tuningcited by 7,869
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  8. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  9. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  10. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
  11. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,188
  12. 2020Concept Bottleneck Modelscited by 1,032
38 more
  1. 2017Network Dissection: Quantifying Interpretability of Deep Visual Representationscited by 708
  2. 2024Scaling and evaluating sparse autoencoderscited by 553
  3. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 326
  4. 2022Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classificationcited by 312
  5. 2024Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoderscited by 300
  6. 2022Post-hoc Concept Bottleneck Modelscited by 293
  7. 2023Label-Free Concept Bottleneck Modelscited by 285
  8. 2023Data Filtering Networkscited by 250
  9. 2025AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoderscited by 218
  10. 2013k-Sparse Autoencoderscited by 215
  11. 2024A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoderscited by 160
  12. 2022CLIP-Dissect: Automatic Description of Neuron Representations in Deep Vision Networkscited by 147
  13. 2024BatchTopK Sparse Autoencoderscited by 130
  14. 2024LLaVA-Phi: Efficient Multi-Modal Assistant with Small Language Modelcited by 129
  15. 2025SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretabilitycited by 124
  16. 2024Towards Interpreting Visual Information Processing in Vision-Language Modelscited by 122
  17. 2024Interpreting CLIP with Sparse Linear Concept Embeddings (SpLiCE)cited by 119
  18. 2025Are Sparse Autoencoders Useful? A Case Study in Sparse Probingcited by 115
  19. 2024Automatically Interpreting Millions of Features in Large Language Modelscited by 113
  20. 2025SAEs Are Good for Steering -- If You Select the Right Featurescited by 78
  21. 2023Learning Concise and Descriptive Attributes for Visual Recognitioncited by 73
  22. 2025Sparse Autoencoders Learn Monosemantic Features in Vision-Language Modelscited by 71
  23. 2024VLG-CBM: Training Concept Bottleneck Models with Vision-Language Guidancecited by 65
  24. 2025Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignmentcited by 62
  25. 2025Interpreting CLIP with Hierarchical Sparse Autoencoderscited by 60
  26. 2018BiHMP-GAN: Bidirectional 3D Human Motion Prediction GANcited by 55
  27. 2024Concept Bottleneck Large Language Modelscited by 43
  28. 2025Interpreting the linear structure of vision-language model embedding spacescited by 32
  29. 2024textitRevelio: Interpreting and leveraging semantic information in diffusion modelscited by 30
  30. 2025LLaVA-MORE: A Comparative Study of LLMs and Visual Backbones for Enhanced Visual Instruction Tuningcited by 28
  31. 2025Interpretable and Testable Vision Features via Sparse Autoencoderscited by 25
  32. 2025How Visual Representations Map to Language Feature Space in Multimodal LLMscited by 25
  33. 2025SAE-V: Interpreting Multimodal Models for Enhanced Alignmentcited by 23
  34. 2025Interpretable Generative Models through Post-hoc Concept Bottleneckscited by 11
  35. 2025Does higher interpretability imply better utility? A Pairwise Analysis on Sparse Autoencoderscited by 10
  36. 2025SPARC: Concept-Aligned Sparse Autoencoders for Cross-Model and Cross-Modal Interpretabilitycited by 9
  37. 2024DECIDER: Leveraging Foundation Model Priors for Improved Model Failure Detection and Explanationcited by 6
  38. 2025AlignSAE: Concept-Aligned Sparse Autoencoderscited by 4

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026Sparse Autoencoders as a Steering Basis for Phase Synchronization in Graph-Based CFD Surrogatescited by 1
  2. 2026Fundamental Limits of Neural Network Sparsification: Evidence from Catastrophic Interpretability Collapsecited by 0
  3. 2026Steered Generation via Gradient-Based Optimization on Sparse Query Featurescited by 0
  4. 2026Multimodal Model Diffing for Feature Discovery and Controlcited by 0
  5. 2026Finding and using interpretable latents in a neutrino foundation model with sparse autoencoderscited by 0
  6. 2026Geometric Similarity in VLM Low-Level Vision Representationscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Projecting Assumptions: The Duality Between Sparse Autoencoders and Concept Geometry
  2. 2024Interpreting Attention Layer Outputs with Sparse Autoencoders
  3. 2026A Geometric View for Understanding Concept Learning and Neuron Interpretation in Sparse Autoencoders
  4. 2024Analyzing (In)Abilities of SAEs via Formal Languages
  5. 2026Supervised sparse auto-encoders for interpretable and compositional representations
  6. 2025Concept-SAE: A Controllable and Invertible Concept Interface for Sparse Autoencoders
  7. 2024SAGE: Scalable Ground Truth Evaluations for Large Sparse Autoencoders
  8. 2026Look But Don't Touch with Sparse Autoencoders for Unlearning in Diffusion Models