# Enhancing Neural Network Interpretability with Feature-Aligned Sparse Autoencoders

arXiv: https://arxiv.org/abs/2411.01220 · HTML: https://mlsanity.com/paper/2411.01220/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Placed in Sparse autoencoder training and evaluation, one of 13 directions in Mechanistic interpretability.
- Builds on 19 of its 41 references that are in the map.
- First posted on arXiv 2 Nov 2024.

## Placed in

- Mechanistic interpretability / [Sparse autoencoder training and evaluation](https://mlsanity.com/mechinterp/sparse-autoencoder-training-and-evaluation/)

## Builds on (most cited first, 19 of 19)

- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels](https://mlsanity.com/paper/1804.06872/) (2018)
- [Temporal Ensembling for Semi-Supervised Learning](https://mlsanity.com/paper/1610.02242/) (2016)
- [Toy Models of Superposition](https://mlsanity.com/paper/2209.10652/) (2022)
- [Network Dissection: Quantifying Interpretability of Deep Visual Representations](https://mlsanity.com/paper/1704.05796/) (2017)
- [Deep Mutual Learning](https://mlsanity.com/paper/1706.00384/) (2017)
- [Scaling and evaluating sparse autoencoders](https://mlsanity.com/paper/2406.04093/) (2024)
- [Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders](https://mlsanity.com/paper/2407.14435/) (2024)
- [k-Sparse Autoencoders](https://mlsanity.com/paper/1312.5663/) (2013)
- [Improving Dictionary Learning with Gated Sparse Autoencoders](https://mlsanity.com/paper/2404.16014/) (2024)
- [Dual Learning for Machine Translation](https://mlsanity.com/paper/1611.00179/) (2016)
- [Deep Co-Training for Semi-Supervised Image Recognition](https://mlsanity.com/paper/1803.05984/) (2018)
- [Multifaceted Feature Visualization: Uncovering the Different Types of Features Learned By Each Neuron in Deep Neural Networks](https://mlsanity.com/paper/1602.03616/) (2016)
- [Dual Student: Breaking the Limits of the Teacher in Semi-supervised Learning](https://mlsanity.com/paper/1909.01804/) (2019)
- [Rigorously Assessing Natural Language Explanations of Neurons](https://mlsanity.com/paper/2309.10312/) (2023)
- [Interpreting Neural Networks through the Polytope Lens](https://mlsanity.com/paper/2211.12312/) (2022)
- [Fraternal Dropout](https://mlsanity.com/paper/1711.00066/) (2017)
- [Engineering Monosemanticity in Toy Models](https://mlsanity.com/paper/2211.09169/) (2022)

## Built on by (most cited first, 25 of 25)

- [A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Models](https://mlsanity.com/paper/2503.05613/) (2025)
- [Position: Mechanistic Interpretability Should Prioritize Feature Consistency in SAEs](https://mlsanity.com/paper/2505.20254/) (2025)
- [Evaluating Adversarial Robustness of Concept Representations in Sparse Autoencoders](https://mlsanity.com/paper/2505.16004/) (2025)
- [Do Sparse Autoencoders Generalize? A Case Study of Answerability](https://mlsanity.com/paper/2502.19964/) (2025)
- [DLM-Scope: Mechanistic Interpretability of Diffusion Language Models via Sparse Autoencoders](https://mlsanity.com/paper/2602.05859/) (2026)
- [LLM Interpretability with Identifiable Temporal-Instantaneous Representation](https://mlsanity.com/paper/2509.23323/) (2025)
- [Unboxing the Black Box: Mechanistic Interpretability for Algorithmic Understanding of Neural Networks](https://mlsanity.com/paper/2511.19265/) (2025)
- [Stable and Steerable Sparse Autoencoders with Weight Regularization](https://mlsanity.com/paper/2603.04198/) (2026)
- [Perplexity Can Miss SAE Feature Damage Under Quantization](https://mlsanity.com/paper/2606.03002/) (2026)
- [Beyond Input Activations: Identifying Influential Latents by Gradient Sparse Autoencoders](https://mlsanity.com/paper/2505.08080/) (2025)
- [AdaptiveK: Complexity-Driven Sparse Autoencoders for Interpretable Language Model Representations](https://mlsanity.com/paper/2508.17320/) (2025)
- [Measuring Sparse Autoencoder Feature Sensitivity](https://mlsanity.com/paper/2509.23717/) (2025)
- [Interpreting CFD Surrogates through Sparse Autoencoders](https://mlsanity.com/paper/2507.16069/) (2025)
- [Visual Exploration of Feature Relationships in Sparse Autoencoders with Curated Concepts](https://mlsanity.com/paper/2511.06048/) (2025)
- [Superposition Without Interference? Towards Isolated Interventions via Almost Orthogonal Features in Language Models](https://mlsanity.com/paper/2602.04718/) (2026)
- [Improving Sparse Autoencoder with Dynamic Attention](https://mlsanity.com/paper/2604.14925/) (2026)
- [Quartet of Diffusions: Structure-Aware Point Cloud Generation through Part and Symmetry Guidance](https://mlsanity.com/paper/2601.20425/) (2026)
- [Sparse Autoencoders as a Steering Basis for Phase Synchronization in Graph-Based CFD Surrogates](https://mlsanity.com/paper/2604.04946/) (2026)
- [Aligned Training: A Parameter-Free Method to Improve Feature Quality and Stability of Sparse Autoencoders (SAE)](https://mlsanity.com/paper/2605.18629/) (2026)
- [A Unifying Framework for Concept-Based Representational Similarity](https://mlsanity.com/paper/2606.09653/) (2026)
- [Concept-SAE: A Controllable and Invertible Concept Interface for Sparse Autoencoders](https://mlsanity.com/paper/2509.22015/) (2025)
- [Query Circuits: Explaining How Language Models Answer User Prompts](https://mlsanity.com/paper/2509.24808/) (2025)
- [Towards Interpretable and Inference-Optimal COT Reasoning with Sparse Autoencoder-Guided Generation](https://mlsanity.com/paper/2510.01528/) (2025)
- [Cross-seed explainability using Procrustes-conditioned Joint End-to-end Top-K Sparse Autoencoders](https://mlsanity.com/paper/2607.08499/) (2026)
- [Feature Evolution and Migration during Vision Transformer Training](https://mlsanity.com/paper/2608.20134/) (2026)

## Abstract (arXiv)

Sparse Autoencoders (SAEs) have shown promise in improving the interpretability of neural network activations, but can learn features that are not features of the input, limiting their effectiveness. We propose textscMutual Feature Regularization (MFR), a regularization technique for improving feature learning by encouraging SAEs trained in parallel to learn similar features. We motivate textscMFR by showing that features learned by multiple SAEs are more likely to correlate with features of the input. By training on synthetic data with known features of the input, we show that textscMFR can help SAEs learn those features, as we can directly compare the features learned by the SAE with the input features for the synthetic data. We then scale textscMFR to SAEs that are trained to denoise electroencephalography (EEG) data and SAEs that are trained to reconstruct GPT-2 Small activations. We show that textscMFR can improve the reconstruction loss of SAEs by up to 21.21% on GPT-2 Small, and 6.67% on EEG data. Our results suggest that the similarity between features learned by different SAEs can be leveraged to improve SAE training, thereby enhancing performance and the usefulness of SAEs for model interpretability.