# On the special role of class-selective neurons in early training

arXiv: https://arxiv.org/abs/2305.17409 · HTML: https://mlsanity.com/paper/2305.17409/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 42 references that are in the map.
- First posted on arXiv 27 May 2023.

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [Network Dissection: Quantifying Interpretability of Deep Visual Representations](https://mlsanity.com/paper/1704.05796/) (2017)
- [Linear Mode Connectivity and the Lottery Ticket Hypothesis](https://mlsanity.com/paper/1912.05671/) (2019)
- [A Benchmark for Interpretability Methods in Deep Neural Networks](https://mlsanity.com/paper/1806.10758/) (2018)
- [Empirical Analysis of the Hessian of Over-Parametrized Neural Networks](https://mlsanity.com/paper/1706.04454/) (2017)
- [Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth](https://mlsanity.com/paper/2010.15327/) (2020)
- [Gradient Descent Happens in a Tiny Subspace](https://mlsanity.com/paper/1812.04754/) (2018)
- [Object Detectors Emerge in Deep Scene CNNs](https://mlsanity.com/paper/1412.6856/) (2014)
- [The Break-Even Point on Optimization Trajectories of Deep Neural Networks](https://mlsanity.com/paper/2002.09572/) (2020)
- [Learning to Generate Reviews and Discovering Sentiment](https://mlsanity.com/paper/1704.01444/) (2017)
- [What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Models](https://mlsanity.com/paper/1812.09355/) (2018)
- [The Early Phase of Neural Network Training](https://mlsanity.com/paper/2002.10365/) (2020)
- [On the importance of single directions for generalization](https://mlsanity.com/paper/1803.06959/) (2018)
- [How Important Is a Neuron?](https://mlsanity.com/paper/1805.12233/) (2018)
- [Ablation Studies in Artificial Neural Networks](https://mlsanity.com/paper/1901.08644/) (2019)
- [Time Matters in Regularizing Deep Networks: Weight Decay and Data Augmentation Affect Early Learning Dynamics, Matter Little Near Convergence](https://mlsanity.com/paper/1905.13277/) (2019)
- [Revisiting the Importance of Individual Units in CNNs via Ablation](https://mlsanity.com/paper/1806.02891/) (2018)
- [Discovery of Natural Language Concepts in Individual Units of CNNs](https://mlsanity.com/paper/1902.07249/) (2019)
- [Lottery Tickets on a Data Diet: Finding Initializations with Sparse Trainable Networks](https://mlsanity.com/paper/2206.01278/) (2022)
- [Selectivity considered harmful: evaluating the causal impact of class selectivity in DNNs](https://mlsanity.com/paper/2003.01262/) (2020)
- [Understanding Neural Networks and Individual Neuron Importance via Information-Ordered Cumulative Ablation](https://mlsanity.com/paper/1804.06679/) (2018)
- [Understanding trained CNNs by indexing neuron selectivity](https://mlsanity.com/paper/1702.00382/) (2017)
- [Deleting object selective units in a fully-connected layer of deep convolutional networks improves classification performance](https://mlsanity.com/paper/2001.07811/) (2020)
- [Linking average- and worst-case perturbation robustness via class selectivity and dimensionality](https://mlsanity.com/paper/2010.07693/) (2020)

## Built on by (most cited first, 5 of 5)

- [Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMs](https://mlsanity.com/paper/2309.07311/) (2023)
- [Training Dynamics of Contextual N-Grams in Language Models](https://mlsanity.com/paper/2311.00863/) (2023)
- [Position: Don't Just "Fix it in Post": A Science of AI Must Study Training Dynamics](https://mlsanity.com/paper/2606.06533/) (2026)
- [Large Learning Rates Simultaneously Achieve Robustness to Spurious Correlations and Compressibility](https://mlsanity.com/paper/2507.17748/) (2025)
- [The Psychology of Falsehood: A Human-Centric Survey of Misinformation Detection](https://mlsanity.com/paper/2509.15896/) (2025)

## Abstract (arXiv)

It is commonly observed that deep networks trained for classification exhibit class-selective neurons in their early and intermediate layers. Intriguingly, recent studies have shown that these class-selective neurons can be ablated without deteriorating network function. But if class-selective neurons are not necessary, why do they exist? We attempt to answer this question in a series of experiments on ResNet-50s trained on ImageNet. We first show that class-selective neurons emerge during the first few epochs of training, before receding rapidly but not completely; this suggests that class-selective neurons found in trained networks are in fact vestigial remains of early training. With single-neuron ablation experiments, we then show that class-selective neurons are important for network function in this early phase of training. We also observe that the network is close to a linear regime in this early phase; we thus speculate that class-selective neurons appear early in training as quasi-linear shortcut solutions to the classification task. Finally, in causal experiments where we regularize against class selectivity at different points in training, we show that the presence of class-selective neurons early in training is critical to the successful training of the network; in contrast, class-selective neurons can be suppressed later in training with little effect on final accuracy. It remains to be understood by which mechanism the presence of class-selective neurons in the early phase of training contributes to the successful training of networks.