mlsanityData to 7 Oct 2026

Supervised sparse auto-encoders for interpretable and compositional representations

arXiv 2602.00924 ↗First posted 31 Jan 2026cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Sparse auto-encoders (SAEs) have re-emerged as a prominent method for mechanistic interpretability, yet they face two significant challenges: the non-smoothness of the L1 penalty, which hinders reconstruction and scalability, and a lack of alignment between learned features and human semantics. In this paper, we address these limitations by adapting unconstrained feature models, a mathematical framework from neural collapse theory, and by supervising the task. We supervise (decoder-only) SAEs to reconstruct feature vectors by jointly learning sparse concept embeddings and decoder weights. Validated on Stable Diffusion 3.5, our approach demonstrates compositional generalization, successfully reconstructing images with concept combinations unseen during training, and enabling feature-level intervention for semantic image editing without prompt modification.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 22 of its 34 references that are in the map. First posted on arXiv 31 Jan 2026.

Builds on

22 of 34 references.

References · 22

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  3. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  4. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  5. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  6. 2016Understanding intermediate layers using linear classifier probescited by 1,279
  7. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,172
  8. 2023Erasing Concepts from Diffusion Modelscited by 530
  9. 2024Scaling and evaluating sparse autoencoderscited by 530
  10. 2024A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoderscited by 148
  11. 2024Interpreting CLIP with Sparse Linear Concept Embeddings (SpLiCE)cited by 116
  12. 2024Towards Principled Evaluations of Sparse Autoencoders for Interpretability and Controlcited by 83
10 more
  1. 2025Sparse Autoencoders Do Not Find Canonical Units of Analysiscited by 80
  2. 2020On the emergence of simplex symmetry in the final and penultimate layers of neural network classifierscited by 51
  3. 2014Sparse Modeling for Image and Vision Processingcited by 48
  4. 2023Deep Neural Collapse Is Provably Optimal for the Deep Unconstrained Features Modelcited by 43
  5. 2024Interpreting Attention Layer Outputs with Sparse Autoencoderscited by 43
  6. 2024One-Step is Enough: Sparse Autoencoders for Text-to-Image Diffusion Modelscited by 30
  7. 2025Sparse Autoencoder as a Zero-Shot Classifier for Concept Erasing in Text-to-Image Diffusion Modelscited by 8
  8. 2025SAEmnesia: Erasing Concepts in Diffusion Models with Supervised Sparse Autoencoderscited by 8
  9. 2025AlignSAE: Concept-Aligned Sparse Autoencoderscited by 4
  10. 2026CASL: Concept-Aligned Sparse Latents for Interpreting Diffusion Modelscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Probing the Representational Power of Sparse Autoencoders in Vision Models
  2. 2026When Structured Sparse Autoencoders Learn Consistent Concepts Across Modalities
  3. 2025Interpreting the linear structure of vision-language model embedding spaces
  4. 2025Sparse Autoencoders, Again?
  5. 2026Evaluating the Interpretability of Sparse Autoencoders with Concept Annotations
  6. 2025Interpreting Large Text-to-Image Diffusion Models with Dictionary Learning
  7. 2025AbsTopK: Rethinking Sparse Autoencoders For Bidirectional Features
  8. 2025Interpreting CLIP with Hierarchical Sparse Autoencoders