mlsanityData to 7 Oct 2026

Transcoders Beat Sparse Autoencoders for Interpretability

arXiv 2501.18823 ↗First posted 31 Jan 2025cs.LGcited by 22

Not on the map. Why?

Abstract (arXiv)

Sparse autoencoders (SAEs) extract human-interpretable features from deep neural networks by transforming their activations into a sparse, higher dimensional latent space, and then reconstructing the activations from these latents. Transcoders are similar to SAEs, but they are trained to reconstruct the output of a component of a deep network given its input. In this work, we compare the features found by transcoders and SAEs trained on the same model and data, finding that transcoder features are significantly more interpretable. We also propose skip transcoders, which add an affine skip connection to the transcoder architecture, and show that these achieve lower reconstruction loss with no effect on interpretability.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). First posted on arXiv 31 Jan 2025.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 10; 2026: 12.

22 papers.

Sort:

By area:

  1. 2025Projecting Assumptions: The Duality Between Sparse Autoencoders and Concept Geometrycited by 58
  2. 2025Towards Understanding Distilled Reasoning Models: A Representational Approachcited by 26
  3. 2025Position: Use Sparse Autoencoders to Discover Unknownscited by 17
  4. 2025Temporal Sparse Autoencoders: Leveraging the Sequential Nature of Language for Interpretabilitycited by 16
  5. 2025Interpretable and Steerable Concept Bottleneck Sparse Autoencoderscited by 6
  6. 2025Towards Interpretability Without Sacrifice: Faithful Dense Layer Decomposition with Mixture of Decoderscited by 5
  7. 2025Binary Sparse Coding for Interpretabilitycited by 3
  8. 2025Beyond External Monitors: Enhancing Transparency of Large Language Models for Easier Monitoringcited by 2
  9. 2025A Unified Theory of Sparse Dictionary Learning in Mechanistic Interpretability: Piecewise Biconvexity and Spurious Minimacited by 2
  10. 2026LLM Layers Immediately Correct Each Othercited by 2
  11. 2026Beyond Dense States: Sparse Transcoders as Causally Testable Operators for LLM Latent Reasoningcited by 1
  12. 2026Improving Robustness In Sparse Autoencoders via Masked Regularizationcited by 1
10 more
  1. 2026WriteSAE: Sparse Autoencoders for Recurrent Statecited by 1
  2. 2025Capacity Matters: a Proof-of-Concept for Transformer Memorization on Real-World Datacited by 0
  3. 2026SCAN: Sparse Circuit Anchor Interpretable Neuron for Lifelong Knowledge Editingcited by 0
  4. 2026Automated Attention Pattern Discovery at Scale in Large Language Modelscited by 0
  5. 2026Geometry-Adaptive Explainer for Faithful Dictionary-Based Interpretability under Distribution Shiftcited by 0
  6. 2026Interactions Between Crosscoder Features: A Compact Proofs Perspectivecited by 0
  7. 2026Decompose Sparsely Where You Should, Absorb Densely Where You Should Nocited by 0
  8. 2026Transcoders for Investigating Deception in Language Modelscited by 0
  9. 2026Emergent Hierarchical Monosemantic Neurons from the Group-Contrastive Forward-Forward Algorithmcited by 0
  10. 2026Intrinsic Structure: Spectral Identifiability for Mechanistic Interpretabilitycited by 0