# Computationally Efficient Measures of Internal Neuron Importance

arXiv: https://arxiv.org/abs/1807.09946 · HTML: https://mlsanity.com/paper/1807.09946/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Jul 2018.

## Built on by (most cited first, 20 of 20)

- [A Primer on the Inner Workings of Transformer-based Language Models](https://mlsanity.com/paper/2405.00208/) (2024)
- [Boosting Multi-modal Model Performance with Adaptive Gradient Modulation](https://mlsanity.com/paper/2308.07686/) (2023)
- [A Rigorous Study of Integrated Gradients Method and Extensions to Internal Neuron Attributions](https://mlsanity.com/paper/2202.11912/) (2022)
- [Software for Dataset-wide XAI: From Local Explanations to Global Insights with Zennit, CoRelAy, and ViRelAy](https://mlsanity.com/paper/2106.13200/) (2021)
- [Language Model Circuits Are Sparse in the Neuron Basis](https://mlsanity.com/paper/2601.22594/) (2026)
- [Restore Anything with Masks: Leveraging Mask Image Modeling for Blind All-in-One Image Restoration](https://mlsanity.com/paper/2409.19403/) (2024)
- [MIMIC-IF: Interpretability and Fairness Evaluation of Deep Learning Models on MIMIC-IV Dataset](https://mlsanity.com/paper/2102.06761/) (2021)
- [Visual Explanations from Deep Networks via Riemann-Stieltjes Integrated Gradient-based Localization](https://mlsanity.com/paper/2205.10900/) (2022)
- [Generalized Attention Flow: Feature Attribution for Transformer Models via Maximum Flow](https://mlsanity.com/paper/2502.15765/) (2025)
- [Deconstructing Pre-training: Knowledge Attribution Analysis in MoE and Dense Models](https://mlsanity.com/paper/2601.08383/) (2026)
- [Time-series attribution maps with regularized contrastive learning](https://mlsanity.com/paper/2502.12977/) (2025)
- [Demystifying Variance in Circuit Discovery of LLMs](https://mlsanity.com/paper/2606.16920/) (2026)
- [ISimDL: Importance Sampling-Driven Acceleration of Fault Injection Simulations for Evaluating the Robustness of Deep Learning](https://mlsanity.com/paper/2303.08035/) (2023)
- [Which Neurons Matter in IR? Applying Integrated Gradients-based Methods to Understand Cross-Encoders](https://mlsanity.com/paper/2406.19309/) (2024)
- [RAM++: Robust Representation Learning via Adaptive Mask for All-in-One Image Restoration](https://mlsanity.com/paper/2509.12039/) (2025)
- [Layer-wise Positional Bias in Short-Context Language Modeling](https://mlsanity.com/paper/2601.04098/) (2026)
- [Matryoshka attribution: Learning to attribute language model outputs to representations and weights](https://mlsanity.com/paper/2609.25518/) (2026)
- [Multimodal Interpretable Data-Driven Models for Early Prediction of Antimicrobial Multidrug Resistance Using Multivariate Time-Series](https://mlsanity.com/paper/2402.06295/) (2024)
- [Y-Drop: A Conductance based Dropout for fully connected layers](https://mlsanity.com/paper/2409.09088/) (2024)
- [Attribution-Based Neuron Utility for Plasticity Restoration in Deep Networks](https://mlsanity.com/paper/2605.06834/) (2026)

## Abstract (arXiv)

The challenge of assigning importance to individual neurons in a network is of interest when interpreting deep learning models. In recent work, Dhamdhere et al. proposed Total Conductance, a "natural refinement of Integrated Gradients" for attributing importance to internal neurons. Unfortunately, the authors found that calculating conductance in tensorflow required the addition of several custom gradient operators and did not scale well. In this work, we show that the formula for Total Conductance is mathematically equivalent to Path Integrated Gradients computed on a hidden layer in the network. We provide a scalable implementation of Total Conductance using standard tensorflow gradient operators that we call Neuron Integrated Gradients. We compare Neuron Integrated Gradients to DeepLIFT, a pre-existing computationally efficient approach that is applicable to calculating internal neuron importance. We find that DeepLIFT produces strong empirical results and is faster to compute, but because it lacks the theoretical properties of Neuron Integrated Gradients, it may not always be preferred in practice. Colab notebook reproducing results: http://bit.ly/neuronintegratedgradients