# Interpreting Deep Neural Networks Through Variable Importance

arXiv: https://arxiv.org/abs/1901.09839 · HTML: https://mlsanity.com/paper/1901.09839/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Jan 2019.

## Built on by (most cited first, 4 of 4)

- [Explanation Strategies as an Empirical-Analytical Lens for Socio-Technical Contextualization of Machine Learning Interpretability](https://mlsanity.com/paper/2109.11849/) (2021)
- [SIA: Symbolic Interpretability for Anticipatory Deep Reinforcement Learning in Network Control](https://mlsanity.com/paper/2601.22044/) (2026)
- [NeuSemSlice: Towards Effective DNN Model Maintenance via Neuron-level Semantic Slicing](https://mlsanity.com/paper/2407.20281/) (2024)
- [Hypothesis Testing and Machine Learning: Interpreting Variable Effects in Deep Artificial Neural Networks using Cohen's f2](https://mlsanity.com/paper/2302.01407/) (2023)

## Abstract (arXiv)

While the success of deep neural networks (DNNs) is well-established across a variety of domains, our ability to explain and interpret these methods is limited. Unlike previously proposed local methods which try to explain particular classification decisions, we focus on global interpretability and ask a universally applicable question: given a trained model, which features are the most important? In the context of neural networks, a feature is rarely important on its own, so our strategy is specifically designed to leverage partial covariance structures and incorporate variable dependence into feature ranking. Our methodological contributions in this paper are two-fold. First, we propose an effect size analogue for DNNs that is appropriate for applications with highly collinear predictors (ubiquitous in computer vision). Second, we extend the recently proposed "RelATive cEntrality" (RATE) measure (Crawford et al., 2019) to the Bayesian deep learning setting. RATE applies an information theoretic criterion to the posterior distribution of effect sizes to assess feature significance. We apply our framework to three broad application areas: computer vision, natural language processing, and social science.