Interpreting and improving deep-learning models with reality checks
Not on the map. Why?
Abstract (arXiv)
Recent deep-learning models have achieved impressive predictive performance by learning complex functions of many variables, often at the cost of interpretability. This chapter covers recent work aiming to interpret models by attributing importance to features and feature groups for a single prediction. Importantly, the proposed attributions assign importance to interactions between features, in addition to features in isolation. These attributions are shown to yield insights across real-world domains, including bio-imaging, cosmology image and natural-language processing. We then show how these attributions can be used to directly improve the generalization of a neural network or to distill it into a simple model. Throughout the chapter, we emphasize the use of reality checks to scrutinize the proposed interpretation techniques.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 47 of its 83 references that are in the map. First posted on arXiv 16 Aug 2021.
Builds on
47 of 83 references.
References · 47
- 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
- 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
- 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
- 2017Axiomatic Attribution for Deep Networkscited by 3,350
- 2013Visualizing and Understanding Convolutional Networkscited by 2,278
- 2017A Survey on Deep Learning in Medical Image Analysiscited by 1,182
- 2018Sanity Checks for Saliency Mapscited by 1,149
- 2014Striving for Simplicity: The All Convolutional Netcited by 983
- 2016The Mythos of Model Interpretabilitycited by 716
- 2019Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)cited by 657
- 2017Interpretable Explanations of Black Boxes by Meaningful Perturbationcited by 611
- 2015Neural Module Networkscited by 375
35 more
- 2016European Union regulations on algorithmic decision-making and a "right to explanation"cited by 343
- 2015Understanding Neural Networks Through Deep Visualizationcited by 332
- 2019Interpretable machine learning: definitions, methods, and applicationscited by 303
- 2016Not Just a Black Box: Learning Important Features Through Propagating Activation Differencescited by 295
- 2017Right for the Right Reasons: Training Differentiable Models by Constraining their Explanationscited by 293
- 2017Word Embeddings Quantify 100 Years of Gender and Ethnic Stereotypescited by 251
- 2017Real Time Image Saliency for Black Box Classifierscited by 246
- 2017Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing their Input Gradientscited by 236
- 2018Women also Snowboard: Overcoming Bias in Captioning Modelscited by 236
- 2017Distilling a Neural Network Into a Soft Decision Treecited by 230
- 2016Understanding Neural Networks through Representation Erasurecited by 223
- 2017Visualizing Deep Neural Network Decisions: Prediction Difference Analysiscited by 166
- 2019REPAIR: Removing Representation Bias by Dataset Resamplingcited by 157
- 2019Interpretations are useful: penalizing explanations to align neural networks with prior knowledgecited by 144
- 2019The Shapley Taylor Interaction Indexcited by 143
- 2015Interpretable classifiers using rules and Bayesian analysis: Building a better stroke prediction modelcited by 140
- 2017Detecting Statistical Interactions from Neural Network Weightscited by 80
- 2019Incorporating Priors with Feature Attribution on Text Classificationcited by 68
- 2018Beyond Word Importance: Contextual Decomposition to Extract Interactions from LSTMscited by 67
- 2018Hierarchical interpretations for neural network predictionscited by 66
- 2018Deriving Machine Attention from Human Rationalescited by 54
- 2017Interpreting CNN Knowledge via an Explanatory Graphcited by 49
- 2021Shapley Explanation Networkscited by 41
- 2018Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Insteadcited by 40
- 2019Embedding Human Knowledge into Deep Neural Network via Attention Mapcited by 33
- 2019Improving performance of deep learning models with axiomatic attribution priors and expected gradientscited by 32
- 2021Adaptive wavelet distillation from neural networks through interpretationscited by 32
- 2019Learning Credible Deep Neural Networks with Rationale Regularizationcited by 18
- 2020Technical Note: Game-Theoretic Interactions of Different Orderscited by 11
- 2017Automatic Rule Extraction from Long Short Term Memory Networkscited by 10
- 2018Can I trust you more? Model-Agnostic Hierarchical Explanationscited by 10
- 2020Transformation Importance with Applications to Cosmologycited by 8
- 2018Learning Sparse Wavelet Representationscited by 6
- 2019Disentangled Attribution Curves for Interpreting Random Forests and Boosted Treescited by 5
- 2019A Simple Saliency Method That Passes the Sanity Checkscited by 2
Built on by
Citing papers per year: 2022: 1.
1 paper.