# One-Step is Enough: Sparse Autoencoders for Text-to-Image Diffusion Models

arXiv: https://arxiv.org/abs/2410.22366 · HTML: https://mlsanity.com/paper/2410.22366/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 28 of its 65 references that are in the map.
- First posted on arXiv 28 Oct 2024.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)
- Mechanistic interpretability / [Interpreting and steering diffusion models](https://mlsanity.com/mechinterp/interpreting-and-steering-diffusion-models/)

## Builds on (most cited first, 28 of 28)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [Steering Llama 2 via Contrastive Activation Addition](https://mlsanity.com/paper/2312.06681/) (2023)
- [Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks](https://mlsanity.com/paper/2401.14159/) (2024)
- [Adversarial Diffusion Distillation](https://mlsanity.com/paper/2311.17042/) (2023)
- [Scaling and evaluating sparse autoencoders](https://mlsanity.com/paper/2406.04093/) (2024)
- [Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models](https://mlsanity.com/paper/2403.19647/) (2024)
- [Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2](https://mlsanity.com/paper/2408.05147/) (2024)
- [Diffusion Models already have a Semantic Latent Space](https://mlsanity.com/paper/2210.10960/) (2022)
- [P+: Extended Textual Conditioning in Text-to-Image Generation](https://mlsanity.com/paper/2303.09522/) (2023)
- [Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models](https://mlsanity.com/paper/2401.06102/) (2024)
- [Concept Sliders: LoRA Adaptors for Precise Control in Diffusion Models](https://mlsanity.com/paper/2311.12092/) (2023)
- [Direct Inversion: Boosting Diffusion-based Editing with 3 Lines of Code](https://mlsanity.com/paper/2310.01506/) (2023)
- [A Practical Review of Mechanistic Interpretability for Transformer-Based Language Models](https://mlsanity.com/paper/2407.02646/) (2024)
- [Transformer visualization via dictionary learning: contextualized embedding as a linear superposition of transformer factors](https://mlsanity.com/paper/2103.15949/) (2021)
- [SelfIE: Self-Interpretation of Large Language Model Embeddings](https://mlsanity.com/paper/2403.10949/) (2024)
- [Wuerstchen: An Efficient Architecture for Large-Scale Text-to-Image Diffusion Models](https://mlsanity.com/paper/2306.00637/) (2023)
- [An Image is Worth Multiple Words: Multi-attribute Inversion for Constrained Text-to-Image Synthesis](https://mlsanity.com/paper/2311.11919/) (2023)

## Built on by (most cited first, 30 of 33)

- [SAeUron: Interpretable Concept Unlearning in Diffusion Models with Sparse Autoencoders](https://mlsanity.com/paper/2501.18052/) (2025)
- [Archetypal SAE: Adaptive and Stable Dictionary Learning for Concept Extraction in Large Vision Models](https://mlsanity.com/paper/2502.12892/) (2025)
- [Sparse Autoencoders Learn Monosemantic Features in Vision-Language Models](https://mlsanity.com/paper/2504.02821/) (2025)
- [Projecting Assumptions: The Duality Between Sparse Autoencoders and Concept Geometry](https://mlsanity.com/paper/2503.01822/) (2025)
- [Interpreting CLIP with Hierarchical Sparse Autoencoders](https://mlsanity.com/paper/2502.20578/) (2025)
- [A Survey on Mechanistic Interpretability for Multi-Modal Foundation Models](https://mlsanity.com/paper/2502.17516/) (2025)
- [Rethinking Evaluation of Sparse Autoencoders through the Representation of Polysemous Words](https://mlsanity.com/paper/2501.06254/) (2025)
- [TIDE : Temporal-Aware Sparse Autoencoders for Interpretable Diffusion Transformers in Image Generation](https://mlsanity.com/paper/2503.07050/) (2025)
- [SAEmnesia: Erasing Concepts in Diffusion Models with Supervised Sparse Autoencoders](https://mlsanity.com/paper/2509.21379/) (2025)
- [DLM-Scope: Mechanistic Interpretability of Diffusion Language Models via Sparse Autoencoders](https://mlsanity.com/paper/2602.05859/) (2026)
- [Interpretable Debiasing of Vision-Language Models for Social Fairness](https://mlsanity.com/paper/2602.24014/) (2026)
- [TADA! Tuning Audio Diffusion Models through Activation Steering](https://mlsanity.com/paper/2602.11910/) (2026)
- [DifFRACT: Diffusion Feature Reconstruction and Attribution for Circuit Tracing](https://mlsanity.com/paper/2606.15796/) (2026)
- [The Many Senses of Visual Similarity: A Text-Prompted Image Perceptual Metric](https://mlsanity.com/paper/2607.18237/) (2026)
- [Beyond Interpretability: When, Why, and How Sparse Autoencoders Enable Label-Free Visual Steering](https://mlsanity.com/paper/2506.01247/) (2025)
- [ELROND: Exploring and decomposing intrinsic capabilities of diffusion models](https://mlsanity.com/paper/2602.10216/) (2026)
- [Language Models Can Explain Visual Features via Steering](https://mlsanity.com/paper/2603.22593/) (2026)
- [SALVE: Sparse Autoencoder-Latent Vector Editing for Mechanistic Control of Neural Networks](https://mlsanity.com/paper/2512.15938/) (2025)
- [Meltdown: Circuits and Bifurcations in Point-Cloud-Conditioned 3D Diffusion Transformers](https://mlsanity.com/paper/2602.11130/) (2026)
- [SoftSAE: Dynamic Top-K Selection for Adaptive Sparse Autoencoders](https://mlsanity.com/paper/2605.06610/) (2026)
- [Residualized Temporal Sparse Autoencoders for Interpreting Diffusion Models](https://mlsanity.com/paper/2605.27813/) (2026)
- [Look But Don't Touch with Sparse Autoencoders for Unlearning in Diffusion Models](https://mlsanity.com/paper/2606.31699/) (2026)
- [Supervised sparse auto-encoders for interpretable and compositional representations](https://mlsanity.com/paper/2602.00924/) (2026)
- [RAIGen: Rare Attribute Identification in Text-to-Image Generative Models](https://mlsanity.com/paper/2602.06806/) (2026)
- [DreamReader: An Interpretability Toolkit for Text-to-Image Models](https://mlsanity.com/paper/2603.13299/) (2026)
- [Can Cross-Layer Transcoders Replace Vision Transformer Activations? An Interpretable Perspective on Vision](https://mlsanity.com/paper/2604.13304/) (2026)
- [Grokking of Diffusion Models: Case Study on Modular Addition](https://mlsanity.com/paper/2604.17673/) (2026)
- [Position: Preventing AI-Generated CSAM Necessitates New Approaches to AI Safety](https://mlsanity.com/paper/2607.05407/) (2026)
- [Interpreting Object-Dependent Concept Brittleness in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2609.09909/) (2026)
- [D-Scope: Decomposing and Steering Diffusion Transformers with Sparse Autoencoders](https://mlsanity.com/paper/2609.39625/) (2026)

## Abstract (arXiv)

For large language models (LLMs), sparse autoencoders (SAEs) have been shown to decompose intermediate representations that often are not interpretable directly into sparse sums of interpretable features, facilitating better control and subsequent analysis. However, similar analyses and approaches have been lacking for text-to-image models. We investigate the possibility of using SAEs to learn interpretable features for SDXL Turbo, a few-step text-to-image diffusion model. To this end, we train SAEs on the updates performed by transformer blocks within SDXL Turbo's denoising U-net in its 1-step setting. Interestingly, we find that they generalize to 4-step SDXL Turbo and even to the multi-step SDXL base model (i.e., a different model) without additional training. In addition, we show that their learned features are interpretable, causally influence the generation process, and reveal specialization among the blocks. We do so by creating RIEBench, a representation-based image editing benchmark, for editing images while they are generated by turning on and off individual SAE features. This allows us to track which transformer blocks' features are the most impactful depending on the edit category. Our work is the first investigation of SAEs for interpretability in text-to-image diffusion models and our results establish SAEs as a promising approach for understanding and manipulating the internal mechanisms of text-to-image models.