# Sparse Autoencoders are Topic Models

arXiv: https://arxiv.org/abs/2511.16309 · HTML: https://mlsanity.com/paper/2511.16309/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 69 of its 108 references that are in the map.
- First posted on arXiv 20 Nov 2025.

## Builds on (most cited first, 30 of 69)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021)
- [InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency](https://mlsanity.com/paper/2508.18265/) (2025)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts](https://mlsanity.com/paper/2102.08981/) (2021)
- [Teaching Machines to Read and Comprehend](https://mlsanity.com/paper/1506.03340/) (2015)
- [Toy Models of Superposition](https://mlsanity.com/paper/2209.10652/) (2022)
- [BERTopic: Neural topic modeling with a class-based TF-IDF procedure](https://mlsanity.com/paper/2203.05794/) (2022)
- [The Linear Representation Hypothesis and the Geometry of Large Language Models](https://mlsanity.com/paper/2311.03658/) (2023)
- [Phi-4 Technical Report](https://mlsanity.com/paper/2412.08905/) (2024)
- [Generating Sentences from a Continuous Space](https://mlsanity.com/paper/1511.06349/) (2015)
- [Scaling and evaluating sparse autoencoders](https://mlsanity.com/paper/2406.04093/) (2024)
- [From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification](https://mlsanity.com/paper/1602.02068/) (2016)
- [Recognize Anything: A Strong Image Tagging Model](https://mlsanity.com/paper/2306.03514/) (2023)
- [Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders](https://mlsanity.com/paper/2407.14435/) (2024)
- [Topic Modeling in Embedding Spaces](https://mlsanity.com/paper/1907.04907/) (2019)
- [AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoders](https://mlsanity.com/paper/2501.17148/) (2025)
- [k-Sparse Autoencoders](https://mlsanity.com/paper/1312.5663/) (2013)
- [Not All Language Model Features Are One-Dimensionally Linear](https://mlsanity.com/paper/2405.14860/) (2024)
- [Autoencoding Variational Inference For Topic Models](https://mlsanity.com/paper/1703.01488/) (2017)
- [Unsupervised Domain Clusters in Pretrained Language Models](https://mlsanity.com/paper/2004.02105/) (2020)
- [BatchTopK Sparse Autoencoders](https://mlsanity.com/paper/2412.06410/) (2024)
- [Top2Vec: Distributed Representations of Topics](https://mlsanity.com/paper/2008.09470/) (2020)
- [SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability](https://mlsanity.com/paper/2503.09532/) (2025)
- [Pre-training is a Hot Topic: Contextualized Document Embeddings Improve Topic Coherence](https://mlsanity.com/paper/2004.03974/) (2020)
- [Neural Variational Inference for Text Processing](https://mlsanity.com/paper/1511.06038/) (2015)
- [Discover-then-Name: Task-Agnostic Concept Bottlenecks via Automated Concept Discovery](https://mlsanity.com/paper/2407.14499/) (2024)

## Built on by (most cited first, 3 of 3)

- [Automated Concept Discovery for LLM-as-a-Judge Preference Analysis](https://mlsanity.com/paper/2603.03319/) (2026)
- [Do Sparse Autoencoders Learn Meaningful Concept Hierarchies?](https://mlsanity.com/paper/2606.22994/) (2026)
- [Beyond Top Words: MonoTM for Topic Modeling with Interpretable Monosemantic Features](https://mlsanity.com/paper/2609.09575/) (2026)

## Abstract (arXiv)

Sparse autoencoders (SAEs) are used to analyze embeddings, but their role and practical value are debated. We propose a new perspective on SAEs by demonstrating that they can be naturally understood as topic models. We propose a continuous topic model (CTM) inspired by Latent Dirichlet Allocation (LDA) for embedding spaces and derive the SAE objective as a maximum a posteriori estimator under this model. This view implies SAE features are thematic components rather than steerable directions. To confirm our theoretical findings, we introduce SAE-TM, a topic modeling framework that: (1) trains an SAE to learn reusable topic atoms, (2) interprets them as word distributions on downstream data, and (3) merges them into any number of topics without retraining. SAE-TM yields more coherent topics than strong baselines on text and image datasets while maintaining diversity. Finally, we analyze thematic structure in image datasets and trace topic changes over time in Japanese woodblock prints. Our work positions SAEs as effective tools for large-scale thematic analysis across modalities. Code is available at https://github.com/ExplainableML/SAE-TM .