# Transcoders Beat Sparse Autoencoders for Interpretability

arXiv: https://arxiv.org/abs/2501.18823 · HTML: https://mlsanity.com/paper/2501.18823/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Placed in Sparse autoencoder training and evaluation, one of 13 directions in Mechanistic interpretability.
- First posted on arXiv 31 Jan 2025.

## Placed in

- Mechanistic interpretability / [Sparse autoencoder training and evaluation](https://mlsanity.com/mechinterp/sparse-autoencoder-training-and-evaluation/)

## Built on by (most cited first, 25 of 25)

- [Verbalizable Representations Form a Global Workspace in Language Models](https://mlsanity.com/paper/2607.15495/) (2026)
- [Projecting Assumptions: The Duality Between Sparse Autoencoders and Concept Geometry](https://mlsanity.com/paper/2503.01822/) (2025)
- [Towards Understanding Distilled Reasoning Models: A Representational Approach](https://mlsanity.com/paper/2503.03730/) (2025)
- [Position: Use Sparse Autoencoders to Discover Unknowns](https://mlsanity.com/paper/2506.23845/) (2025)
- [Temporal Sparse Autoencoders: Leveraging the Sequential Nature of Language for Interpretability](https://mlsanity.com/paper/2511.05541/) (2025)
- [Interpretable and Steerable Concept Bottleneck Sparse Autoencoders](https://mlsanity.com/paper/2512.10805/) (2025)
- [Towards Interpretability Without Sacrifice: Faithful Dense Layer Decomposition with Mixture of Decoders](https://mlsanity.com/paper/2505.21364/) (2025)
- [Binary Sparse Coding for Interpretability](https://mlsanity.com/paper/2509.25596/) (2025)
- [A Unified Theory of Sparse Dictionary Learning in Mechanistic Interpretability: Piecewise Biconvexity and Spurious Minima](https://mlsanity.com/paper/2512.05534/) (2025)
- [LLM Layers Immediately Correct Each Other](https://mlsanity.com/paper/2609.07876/) (2026)
- [Beyond External Monitors: Enhancing Transparency of Large Language Models for Easier Monitoring](https://mlsanity.com/paper/2502.05242/) (2025)
- [Beyond Dense States: Sparse Transcoders as Causally Testable Operators for LLM Latent Reasoning](https://mlsanity.com/paper/2602.01695/) (2026)
- [Improving Robustness In Sparse Autoencoders via Masked Regularization](https://mlsanity.com/paper/2604.06495/) (2026)
- [WriteSAE: Sparse Autoencoders for Recurrent State](https://mlsanity.com/paper/2605.12770/) (2026)
- [Capacity Matters: a Proof-of-Concept for Transformer Memorization on Real-World Data](https://mlsanity.com/paper/2506.14704/) (2025)
- [SCAN: Sparse Circuit Anchor Interpretable Neuron for Lifelong Knowledge Editing](https://mlsanity.com/paper/2603.15226/) (2026)
- [Automated Attention Pattern Discovery at Scale in Large Language Models](https://mlsanity.com/paper/2604.03764/) (2026)
- [Geometry-Adaptive Explainer for Faithful Dictionary-Based Interpretability under Distribution Shift](https://mlsanity.com/paper/2605.21849/) (2026)
- [Interactions Between Crosscoder Features: A Compact Proofs Perspective](https://mlsanity.com/paper/2606.09940/) (2026)
- [Decompose Sparsely Where You Should, Absorb Densely Where You Should No](https://mlsanity.com/paper/2606.14040/) (2026)
- [Transcoders for Investigating Deception in Language Models](https://mlsanity.com/paper/2607.14791/) (2026)
- [Emergent Hierarchical Monosemantic Neurons from the Group-Contrastive Forward-Forward Algorithm](https://mlsanity.com/paper/2607.16295/) (2026)
- [Intrinsic Structure: Spectral Identifiability for Mechanistic Interpretability](https://mlsanity.com/paper/2608.10172/) (2026)
- [Sparse Transformer: a Controlled Testbed for Mechanistic Interpretability](https://mlsanity.com/paper/2610.19907/) (2026)
- [Sparse Autoencoders for Mechanistic Interpretability in NLP: A Critical Survey of Recoverability, Interpretability, and Utility](https://mlsanity.com/paper/2610.20139/) (2026)

## Abstract (arXiv)

Sparse autoencoders (SAEs) extract human-interpretable features from deep neural networks by transforming their activations into a sparse, higher dimensional latent space, and then reconstructing the activations from these latents. Transcoders are similar to SAEs, but they are trained to reconstruct the output of a component of a deep network given its input. In this work, we compare the features found by transcoders and SAEs trained on the same model and data, finding that transcoder features are significantly more interpretable. We also propose skip transcoders, which add an affine skip connection to the transcoder architecture, and show that these achieve lower reconstruction loss with no effect on interpretability.