# How Important Is a Neuron?

arXiv: https://arxiv.org/abs/1805.12233 · HTML: https://mlsanity.com/paper/1805.12233/

## Key facts

- Cited by 73 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 May 2018.

## Built on by (most cited first, 30 of 73)

- [Finding Neurons in a Haystack: Case Studies with Sparse Probing](https://mlsanity.com/paper/2305.01610/) (2023)
- [Explaining in Style: Training a GAN to explain a classifier in StyleSpace](https://mlsanity.com/paper/2104.13369/) (2021)
- [Improving Adversarial Transferability via Neuron Attribution-Based Attacks](https://mlsanity.com/paper/2204.00008/) (2022)
- [A Primer on the Inner Workings of Transformer-based Language Models](https://mlsanity.com/paper/2405.00208/) (2024)
- [Neuron-level Interpretation of Deep NLP Models: A Survey](https://mlsanity.com/paper/2108.13138/) (2021)
- [A Rigorous Study of Integrated Gradients Method and Extensions to Internal Neuron Attributions](https://mlsanity.com/paper/2202.11912/) (2022)
- [Software for Dataset-wide XAI: From Local Explanations to Global Insights with Zennit, CoRelAy, and ViRelAy](https://mlsanity.com/paper/2106.13200/) (2021)
- [Sharing Matters: Analysing Neurons Across Languages and Tasks in LLMs](https://mlsanity.com/paper/2406.09265/) (2024)
- [Finding Discriminative Filters for Specific Degradations in Blind Super-Resolution](https://mlsanity.com/paper/2108.01070/) (2021)
- [Optimal ablation for interpretability](https://mlsanity.com/paper/2409.09951/) (2024)
- [Language Model Circuits Are Sparse in the Neuron Basis](https://mlsanity.com/paper/2601.22594/) (2026)
- [FedClassAvg: Local Representation Learning for Personalized Federated Learning on Heterogeneous Neural Networks](https://mlsanity.com/paper/2210.14226/) (2022)
- [Robust Models Are More Interpretable Because Attributions Look Normal](https://mlsanity.com/paper/2103.11257/) (2021)
- [Discovering Salient Neurons in Deep NLP Models](https://mlsanity.com/paper/2206.13288/) (2022)
- [Restore Anything with Masks: Leveraging Mask Image Modeling for Blind All-in-One Image Restoration](https://mlsanity.com/paper/2409.19403/) (2024)
- [Decomposing and Editing Predictions by Modeling Model Computation](https://mlsanity.com/paper/2404.11534/) (2024)
- [Truly Sparse Neural Networks at Scale](https://mlsanity.com/paper/2102.01732/) (2021)
- [The Manifold Hypothesis for Gradient-Based Explanations](https://mlsanity.com/paper/2206.07387/) (2022)
- [MIMIC-IF: Interpretability and Fairness Evaluation of Deep Learning Models on MIMIC-IV Dataset](https://mlsanity.com/paper/2102.06761/) (2021)
- [Cartoon Explanations of Image Classifiers](https://mlsanity.com/paper/2110.03485/) (2021)
- [Debiasing Deep Chest X-Ray Classifiers using Intra- and Post-processing Methods](https://mlsanity.com/paper/2208.00781/) (2022)
- [A Unified Paths Perspective for Pruning at Initialization](https://mlsanity.com/paper/2101.10552/) (2021)
- [RoCourseNet: Distributionally Robust Training of a Prediction Aware Recourse Model](https://mlsanity.com/paper/2206.00700/) (2022)
- [NeuroX Library for Neuron Analysis of Deep NLP Models](https://mlsanity.com/paper/2305.17073/) (2023)
- [Continuous Human Action Recognition for Human-Machine Interaction: A Review](https://mlsanity.com/paper/2202.13096/) (2022)
- [Evaluating Neuron Interpretation Methods of NLP Models](https://mlsanity.com/paper/2301.12608/) (2023)
- [Discovering Influential Neuron Path in Vision Transformers](https://mlsanity.com/paper/2503.09046/) (2025)
- [Deeply Explain CNN via Hierarchical Decomposition](https://mlsanity.com/paper/2201.09205/) (2022)
- [Adapting to Non-Centered Languages for Zero-shot Multilingual Translation](https://mlsanity.com/paper/2209.04138/) (2022)
- [Boosting Adversarial Transferability via Fusing Logits of Top-1 Decomposed Feature](https://mlsanity.com/paper/2305.01361/) (2023)

## Abstract (arXiv)

The problem of attributing a deep network's prediction to its input/base features is well-studied. We introduce the notion of conductance to extend the notion of attribution to the understanding the importance of hidden units. Informally, the conductance of a hidden unit of a deep network is the flow of attribution via this hidden unit. We use conductance to understand the importance of a hidden unit to the prediction for a specific input, or over a set of inputs. We evaluate the effectiveness of conductance in multiple ways, including theoretical properties, ablation studies, and a feature selection task. The empirical evaluations are done using the Inception network over ImageNet data, and a sentiment analysis network over reviews. In both cases, we demonstrate the effectiveness of conductance in identifying interesting insights about the internal workings of these networks.