# Under the Hood of Neural Networks: Characterizing Learned Representations by Functional Neuron Populations and Network Ablations

arXiv: https://arxiv.org/abs/2004.01254 · HTML: https://mlsanity.com/paper/2004.01254/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Apr 2020.

## Built on by (most cited first, 12 of 12)

- [Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks](https://mlsanity.com/paper/2207.13243/) (2022)
- [Modular Deep Learning](https://mlsanity.com/paper/2302.11529/) (2023)
- [Neuron-level Interpretation of Deep NLP Models: A Survey](https://mlsanity.com/paper/2108.13138/) (2021)
- [Contrastive Explanations for Model Interpretability](https://mlsanity.com/paper/2103.01378/) (2021)
- [Differentiation and Specialization of Attention Heads via the Refined Local Learning Coefficient](https://mlsanity.com/paper/2410.02984/) (2024)
- [How to Certify Machine Learning Based Safety-critical Systems? A Systematic Literature Review](https://mlsanity.com/paper/2107.12045/) (2021)
- [Discovering Salient Neurons in Deep NLP Models](https://mlsanity.com/paper/2206.13288/) (2022)
- [Dynamics of specialization in neural modules under resource constraints](https://mlsanity.com/paper/2106.02626/) (2021)
- [R2U3D: Recurrent Residual 3D U-Net for Lung Segmentation](https://mlsanity.com/paper/2105.02290/) (2021)
- [Future Artificial Intelligence tools and perspectives in medicine](https://mlsanity.com/paper/2206.03289/) (2022)
- [Much Easier Said Than Done: Falsifying the Causal Relevance of Linear Decoding Methods](https://mlsanity.com/paper/2211.04367/) (2022)
- [Modularity in Transformers: Investigating Neuron Separability & Specialization](https://mlsanity.com/paper/2408.17324/) (2024)

## Abstract (arXiv)

The need for more transparency of the decision-making processes in artificial neural networks steadily increases driven by their applications in safety critical and ethically challenging domains such as autonomous driving or medical diagnostics. We address today's lack of transparency of neural networks and shed light on the roles of single neurons and groups of neurons within the network fulfilling a learned task. Inspired by research in the field of neuroscience, we characterize the learned representations by activation patterns and network ablations, revealing functional neuron populations that a) act jointly in response to specific stimuli or b) have similar impact on the network's performance after being ablated. We find that neither a neuron's magnitude or selectivity of activation, nor its impact on network performance are sufficient stand-alone indicators for its importance for the overall task. We argue that such indicators are essential for future advances in transfer learning and modern neuroscience.