mlsanityData to 14 Oct 2026

Deriving Decoder-Free Sparse Autoencoders from First Principles

arXiv 2601.06478 ↗First posted 10 Jan 2026cs.LG stat.MLcited by 0

On the map: Mechanistic interpretability → Sparse autoencoder training and evaluation

Abstract (arXiv)

Gradient descent on log-sum-exp (LSE) objectives performs implicit expectation--maximization (EM): the gradient with respect to each component output equals its responsibility. The same theory predicts collapse without volume control analogous to the log-determinant in Gaussian mixture models. We instantiate the theory in a single-layer encoder with an LSE objective and InfoMax regularization for volume control. Experiments confirm the theory's predictions. The gradient--responsibility identity holds exactly; LSE alone collapses; variance prevents dead components; decorrelation prevents redundancy. The model exhibits EM-like optimization dynamics in which lower loss does not correspond to better features and adaptive optimizers offer no advantage. The resulting decoder-free model learns interpretable mixture components, confirming that implicit EM theory can prescribe architectures.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Sparse autoencoder training and evaluation, one of 13 directions in Mechanistic interpretability. Builds on 5 of its 18 references that are in the map. First posted on arXiv 10 Jan 2026.

Builds on

5 of 18 references.

References · 5

  1. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  2. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
  3. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
  4. 2024Interpreting Neural Networks through Mahalanobis Distancecited by 3
  5. 2025Gradient Descent as Implicit EM in Distance-Based Neural Modelscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Entropic Auto-Encoding via Implicit Free-Energy Minimization
  2. 2024Compression of Structured Data with Autoencoders: Provable Benefit of Nonlinearities and Depth
  3. 2026Expander Sparse Autoencoders: Parameter-Efficient Dictionaries for Mechanistic Interpretability
  4. 2026Subspace-Aware Sparse Autoencoders for Effective Mechanistic Interpretability
  5. 2026Toward Identifiable Sparse Autoencoders
  6. 2026Inference and learning in sparse autoencoders as natural gradient flow
  7. 2026The Diffusion Encoder
  8. 2026Decoder-Preserving Sparse Autoencoders: Which Readouts Survive Sparse Compression?