mlsanityData to 14 Oct 2026

Kronecker Factorization Improves Efficiency and Interpretability of Sparse Autoencoders

arXiv 2505.22255 ↗First posted 28 May 2025cs.LG cs.CLcited by 0

On the map: Mechanistic interpretability → SAE features and their applications

Abstract (arXiv)

Sparse Autoencoders (SAEs) decompose language-model activations into sparse, interpretable features, but standard encoders usually treat the latent dictionary as a flat set of independent coordinates, leaving hierarchy and feature interactions to emerge only implicitly. We propose KronSAE, a design that factorizes the latent space into heads and forms post-latent features as pairwise compositions of lower-dimensional pre-latents using mAND, a differentiable AND-like interaction. This imposes a compositional co-activation prior while remaining compatible with standard SAE objectives and variants such as TopK, Matryoshka, and Switch SAEs. KronSAE matches strong baselines on the EV-FLOPs frontier, improves interpretability of the latents, better captures the underlying correlated feature structure, and reduces encoder computational cost as an additional benefit. Code is available at https://github.com/corl-team/kronsae.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in SAE features and their applications, one of 13 directions in Mechanistic interpretability. Builds on 27 of its 45 references that are in the map. First posted on arXiv 28 May 2025.

Builds on

27 of 45 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability12 Mar
  2. 2025Sparse Autoencoders Trained on the Same Data Learn Different Features28 Jan
  3. 2024Automatically Interpreting Millions of Features in Large Language Models17 Oct
  4. 2024A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders22 Sep
  5. 2024Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders19 Jul
  6. 2024Scaling and evaluating sparse autoencoders6 Jun
  7. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Models15 Sep

Other references · 20

  1. 2024Qwen2.5 Technical Reportcited by 4,942
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  3. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  4. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  5. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  6. 2022Toy Models of Superpositioncited by 971
  7. 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 435
  8. 2024Transcoders Find Interpretable LLM Feature Circuitscited by 200
  9. 2025Learning Multi-Level Features with Matryoshka Sparse Autoencoderscited by 143
  10. 2022Monarch: Expressive Structured Matrices for Efficient and Accurate Trainingcited by 106
  11. 2023Learning to Grow Pretrained Models for Efficient Transformer Trainingcited by 83
  12. 2025Archetypal SAE: Adaptive and Stable Dictionary Learning for Concept Extraction in Large Vision Modelscited by 73
8 more
  1. 2024Efficient Dictionary Learning with Switch Sparse Autoencoderscited by 39
  2. 2019Discovering the Compositional Structure of Vector Representations with Role Learning Networkscited by 34
  3. 2025Automated Interpretability Metrics Do Not Distinguish Trained and Random Transformerscited by 30
  4. 2021Kronecker Decomposition for GPT Compressioncited by 28
  5. 2024Mechanistic Permutability: Match Features Across Layerscited by 22
  6. 2024Compute Optimal Inference and Provable Amortisation Gap in Sparse Autoencoderscited by 18
  7. 2024Mechanisms of Symbol Processing for In-Context Learning in Transformer Networkscited by 6
  8. 2021Logical Activation Functions: Logit-space equivalents of Probabilistic Boolean Operatorscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Beyond Input Activations: Identifying Influential Latents by Gradient Sparse Autoencoders
  2. 2025A Comparative Analysis of Sparse Autoencoder and Activation Difference in Language Model Steering
  3. 2026Subspace-Aware Sparse Autoencoders for Effective Mechanistic Interpretability
  4. 2025SplInterp: Improving our Understanding and Training of Sparse Autoencoders
  5. 2025Evaluating and Designing Sparse Autoencoders by Approximating Quasi-Orthogonality
  6. 2026Discovering Millions of Interpretable Features with Sparse Autoencoders
  7. 2025Rethinking Sparse Autoencoders: Select-and-Project for Fairness and Control from Encoder Features Alone
  8. 2025Revisiting End-To-End Sparse Autoencoder Training: A Short Finetune Is All You Need