mlsanityData to 14 Oct 2026

Enhancing Neural Network Interpretability with Feature-Aligned Sparse Autoencoders

arXiv 2411.01220 ↗First posted 2 Nov 2024cs.LGcited by 25

On the map: Mechanistic interpretability → Sparse autoencoder training and evaluation

Abstract (arXiv)

Sparse Autoencoders (SAEs) have shown promise in improving the interpretability of neural network activations, but can learn features that are not features of the input, limiting their effectiveness. We propose textscMutual Feature Regularization (MFR), a regularization technique for improving feature learning by encouraging SAEs trained in parallel to learn similar features. We motivate textscMFR by showing that features learned by multiple SAEs are more likely to correlate with features of the input. By training on synthetic data with known features of the input, we show that textscMFR can help SAEs learn those features, as we can directly compare the features learned by the SAE with the input features for the synthetic data. We then scale textscMFR to SAEs that are trained to denoise electroencephalography (EEG) data and SAEs that are trained to reconstruct GPT-2 Small activations. We show that textscMFR can improve the reconstruction loss of SAEs by up to 21.21% on GPT-2 Small, and 6.67% on EEG data. Our results suggest that the similarity between features learned by different SAEs can be leveraged to improve SAE training, thereby enhancing performance and the usefulness of SAEs for model interpretability.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Placed in Sparse autoencoder training and evaluation, one of 13 directions in Mechanistic interpretability. Builds on 19 of its 41 references that are in the map. First posted on arXiv 2 Nov 2024.

Builds on

19 of 41 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Toy Models of Superposition21 Sep

Other references · 18

  1. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  2. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
  3. 2018Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labelscited by 1,059
  4. 2016Temporal Ensembling for Semi-Supervised Learningcited by 1,057
  5. 2017Network Dissection: Quantifying Interpretability of Deep Visual Representationscited by 708
  6. 2017Deep Mutual Learningcited by 628
  7. 2024Scaling and evaluating sparse autoencoderscited by 553
  8. 2024Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoderscited by 300
  9. 2013k-Sparse Autoencoderscited by 215
  10. 2024Improving Dictionary Learning with Gated Sparse Autoencoderscited by 179
  11. 2016Dual Learning for Machine Translationcited by 148
  12. 2018Deep Co-Training for Semi-Supervised Image Recognitioncited by 147
6 more
  1. 2016Multifaceted Feature Visualization: Uncovering the Different Types of Features Learned By Each Neuron in Deep Neural Networkscited by 106
  2. 2019Dual Student: Breaking the Limits of the Teacher in Semi-supervised Learningcited by 104
  3. 2023Rigorously Assessing Natural Language Explanations of Neuronscited by 45
  4. 2022Interpreting Neural Networks through the Polytope Lenscited by 33
  5. 2017Fraternal Dropoutcited by 14
  6. 2022Engineering Monosemanticity in Toy Modelscited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 14; 2026: 11.

25 papers.

Sort:

By area:

  1. 2025A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Modelscited by 76
  2. 2025Position: Mechanistic Interpretability Should Prioritize Feature Consistency in SAEscited by 14
  3. 2025Evaluating Adversarial Robustness of Concept Representations in Sparse Autoencoderscited by 12
  4. 2025Do Sparse Autoencoders Generalize? A Case Study of Answerabilitycited by 8
  5. 2026DLM-Scope: Mechanistic Interpretability of Diffusion Language Models via Sparse Autoencoderscited by 7
  6. 2025LLM Interpretability with Identifiable Temporal-Instantaneous Representationcited by 6
  7. 2025Unboxing the Black Box: Mechanistic Interpretability for Algorithmic Understanding of Neural Networkscited by 6
  8. 2026Stable and Steerable Sparse Autoencoders with Weight Regularizationcited by 5
  9. 2026Perplexity Can Miss SAE Feature Damage Under Quantizationcited by 5
  10. 2025Beyond Input Activations: Identifying Influential Latents by Gradient Sparse Autoencoderscited by 4
  11. 2025AdaptiveK: Complexity-Driven Sparse Autoencoders for Interpretable Language Model Representationscited by 4
  12. 2025Measuring Sparse Autoencoder Feature Sensitivitycited by 4
13 more
  1. 2025Interpreting CFD Surrogates through Sparse Autoencoderscited by 3
  2. 2025Visual Exploration of Feature Relationships in Sparse Autoencoders with Curated Conceptscited by 2
  3. 2026Superposition Without Interference? Towards Isolated Interventions via Almost Orthogonal Features in Language Modelscited by 2
  4. 2026Improving Sparse Autoencoder with Dynamic Attentioncited by 2
  5. 2026Quartet of Diffusions: Structure-Aware Point Cloud Generation through Part and Symmetry Guidancecited by 1
  6. 2026Sparse Autoencoders as a Steering Basis for Phase Synchronization in Graph-Based CFD Surrogatescited by 1
  7. 2026Aligned Training: A Parameter-Free Method to Improve Feature Quality and Stability of Sparse Autoencoders (SAE)cited by 1
  8. 2026A Unifying Framework for Concept-Based Representational Similaritycited by 1
  9. 2025Concept-SAE: A Controllable and Invertible Concept Interface for Sparse Autoencoderscited by 0
  10. 2025Query Circuits: Explaining How Language Models Answer User Promptscited by 0
  11. 2025Towards Interpretable and Inference-Optimal COT Reasoning with Sparse Autoencoder-Guided Generationcited by 0
  12. 2026Cross-seed explainability using Procrustes-conditioned Joint End-to-end Top-K Sparse Autoencoderscited by 0
  13. 2026Feature Evolution and Migration during Vision Transformer Trainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Improving Robustness In Sparse Autoencoders via Masked Regularization
  2. 2024Interpreting Attention Layer Outputs with Sparse Autoencoders
  3. 2026Sanity Checks for Sparse Autoencoders: Do SAEs Beat Random Baselines?
  4. 2025Evaluating Sparse Autoencoders: From Shallow Design to Matching Pursuit
  5. 2025SplInterp: Improving our Understanding and Training of Sparse Autoencoders
  6. 2025Binary Sparse Coding for Interpretability
  7. 2024Features that Make a Difference: Leveraging Gradients for Improved Dictionary Learning
  8. 2026AudioSAE: Towards Understanding of Audio-Processing Models with Sparse AutoEncoders