# Understanding trained CNNs by indexing neuron selectivity

arXiv: https://arxiv.org/abs/1702.00382 · HTML: https://mlsanity.com/paper/1702.00382/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Feb 2017.

## Built on by (most cited first, 12 of 12)

- [Neither hype nor gloom do DNNs justice](https://mlsanity.com/paper/2312.05355/) (2023)
- [Towards Interpretable Deep Networks for Monocular Depth Estimation](https://mlsanity.com/paper/2108.05312/) (2021)
- [Causal Analysis for Robust Interpretability of Neural Networks](https://mlsanity.com/paper/2305.08950/) (2023)
- [Explaining Classifiers by Constructing Familiar Concepts](https://mlsanity.com/paper/2203.04109/) (2022)
- [Color Equivariant Convolutional Networks](https://mlsanity.com/paper/2310.19368/) (2023)
- [On the special role of class-selective neurons in early training](https://mlsanity.com/paper/2305.17409/) (2023)
- [Beyond the limitations of any imaginable mechanism: large language models and psycholinguistics](https://mlsanity.com/paper/2303.00077/) (2023)
- [Scalable Visual Attribute Extraction through Hidden Layers of a Residual ConvNet](https://mlsanity.com/paper/2104.00161/) (2021)
- [Canoe : A System for Collaborative Learning for Neural Nets](https://mlsanity.com/paper/2108.12124/) (2021)
- [FICNN: A Framework for the Interpretation of Deep Convolutional Neural Networks](https://mlsanity.com/paper/2305.10121/) (2023)
- [Fixing the problems of deep neural networks will require better training data and learning algorithms](https://mlsanity.com/paper/2311.12819/) (2023)
- [Deep Model Interpretation with Limited Data : A Coreset-based Approach](https://mlsanity.com/paper/2410.00524/) (2024)

## Abstract (arXiv)

The impressive performance of Convolutional Neural Networks (CNNs) when solving different vision problems is shadowed by their black-box nature and our consequent lack of understanding of the representations they build and how these representations are organized. To help understanding these issues, we propose to describe the activity of individual neurons by their Neuron Feature visualization and quantify their inherent selectivity with two specific properties. We explore selectivity indexes for: an image feature (color); and an image label (class membership). Our contribution is a framework to seek or classify neurons by indexing on these selectivity properties. It helps to find color selective neurons, such as a red-mushroom neuron in layer Conv4 or class selective neurons such as dog-face neurons in layer Conv5 in VGG-M, and establishes a methodology to derive other selectivity properties. Indexing on neuron selectivity can statistically draw how features and classes are represented through layers in a moment when the size of trained nets is growing and automatic tools to index neurons can be helpful.