mlsanityData to 14 Oct 2026

Enforcing Orderedness to Improve Feature Consistency

arXiv 2512.02194 ↗First posted 1 Dec 2025cs.LG cs.AIcited by 1

On the map: Mechanistic interpretability → Sparse autoencoder training and evaluation

Abstract (arXiv)

Sparse autoencoders (SAEs) have been widely used for interpretability of neural networks, but their learned features often vary across seeds and hyperparameter settings. We introduce Ordered Sparse Autoencoders (OSAE), which extend Matryoshka SAEs by (1) establishing a strict ordering of latent features and (2) deterministically using every feature dimension, avoiding the sampling-based approximations of prior nested SAE methods. Theoretically, we show that OSAEs resolve permutation non-identifiability in settings of sparse dictionary learning where solutions are unique (up to natural symmetries). Empirically on Gemma2-2B and Pythia-70M, we show that OSAEs can help improve consistency compared to Matryoshka baselines.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Sparse autoencoder training and evaluation, one of 13 directions in Mechanistic interpretability. Builds on 10 of its 23 references that are in the map. First posted on arXiv 1 Dec 2025.

Builds on

10 of 23 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Position: Mechanistic Interpretability Should Prioritize Feature Consistency in SAEs26 May
  2. 2025Evaluating and Designing Sparse Autoencoders by Approximating Quasi-Orthogonality31 Mar
  3. 2025Learning Multi-Level Features with Matryoshka Sparse Autoencoders21 Mar
  4. 2025Archetypal SAE: Adaptive and Stable Dictionary Learning for Concept Extraction in Large Vision Models18 Feb
  5. 2025Sparse Autoencoders Do Not Find Canonical Units of Analysis7 Feb

Other references · 5

  1. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  2. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  3. 2020Neuron Shapley: Discovering the Responsible Neuronscited by 98
  4. 2014Learning Ordered Representations with Nested Dropoutcited by 84
  5. 2018From Principal Subspaces to Principal Components with Linear Autoencoderscited by 39

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Unstable Features, Reproducible Subspaces: Understanding Seed Dependence in Sparse Autoencoderscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026Toward Identifiable Sparse Autoencoders
  2. 2025SplInterp: Improving our Understanding and Training of Sparse Autoencoders
  3. 2025Empirical Evaluation of Progressive Coding for Sparse Autoencoders
  4. 2025Equivariant Sparse Autoencoders: Mechanistic Interpretability of Neural Networks on Symmetric Data
  5. 2025Evaluating Sparse Autoencoders: From Shallow Design to Matching Pursuit
  6. 2026Sanity Checks for Sparse Autoencoders: Do SAEs Beat Random Baselines?
  7. 2025Sparse Shift Autoencoders for Identifying Concepts from Large Language Model Activations
  8. 2026Feature Starvation as Geometric Instability in Sparse Autoencoders