# SAFR: Neuron Redistribution for Interpretability

arXiv: https://arxiv.org/abs/2501.16374 · HTML: https://mlsanity.com/paper/2501.16374/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 31 references that are in the map.
- First posted on arXiv 23 Jan 2025.

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Toy Models of Superposition](https://mlsanity.com/paper/2209.10652/) (2022)
- [Attention is not Explanation](https://mlsanity.com/paper/1902.10186/) (2019)
- [Finding Neurons in a Haystack: Case Studies with Sparse Probing](https://mlsanity.com/paper/2305.01610/) (2023)
- [Linear Algebraic Structure of Word Senses, with Applications to Polysemy](https://mlsanity.com/paper/1601.03764/) (2016)
- [Learning Variational Word Masks to Improve the Interpretability of Neural Text Classifiers](https://mlsanity.com/paper/2010.00667/) (2020)
- [Mathematical Models of Computation in Superposition](https://mlsanity.com/paper/2408.05451/) (2024)
- [Interpreting Neural Networks through the Polytope Lens](https://mlsanity.com/paper/2211.12312/) (2022)
- [Dynamical versus Bayesian Phase Transitions in a Toy Model of Superposition](https://mlsanity.com/paper/2310.06301/) (2023)
- [PURE: Turning Polysemantic Neurons Into Pure Features by Identifying Relevant Circuits](https://mlsanity.com/paper/2404.06453/) (2024)
- [What Causes Polysemanticity? An Alternative Origin Story of Mixed Selectivity from Incidental Causes](https://mlsanity.com/paper/2312.03096/) (2023)
- [Understanding polysemanticity in neural networks through coding theory](https://mlsanity.com/paper/2401.17975/) (2024)
- [Few-shot Named Entity Recognition via Superposition Concept Discrimination](https://mlsanity.com/paper/2403.16463/) (2024)
- [On Implications of Scaling Laws on Feature Superposition](https://mlsanity.com/paper/2407.01459/) (2024)

## Built on by (most cited first, 2 of 2)

- [Signal in the Noise: Polysemantic Interference Transfers and Predicts Cross-Model Influence](https://mlsanity.com/paper/2505.11611/) (2025)
- [Steering Information Utility in Key-Value Memory for Language Model Post-Training](https://mlsanity.com/paper/2507.05158/) (2025)

## Abstract (arXiv)

Superposition refers to encoding representations of multiple features within a single neuron, which is common in deep neural networks. This property allows neurons to combine and represent multiple features, enabling the model to capture intricate information and handle complex tasks. Despite promising performance, the model's interpretability has been diminished. This paper presents a novel approach to enhance model interpretability by regularizing feature superposition. We introduce SAFR, which simply applies regularizations to the loss function to promote monosemantic representations for important tokens while encouraging polysemanticity for correlated token pairs, where important tokens and correlated token pairs are identified via VMASK and attention weights respectively. We evaluate SAFR with a transformer model on two classification tasks. Experiments demonstrate the effectiveness of SAFR in improving model interpretability without compromising prediction performance. Besides, SAFR provides explanations by visualizing the neuron allocation within the intermediate layers.