Explaining Natural Language Processing Classifiers with Occlusion and Language Modeling
Not on the map. Why?
Abstract (arXiv)
Deep neural networks are powerful statistical learners. However, their predictions do not come with an explanation of their process. To analyze these models, explanation methods are being developed. We present a novel explanation method, called OLM, for natural language processing classifiers. This method combines occlusion and language modeling, which are techniques central to explainability and NLP, respectively. OLM gives explanations that are theoretically sound and easy to understand. We make several contributions to the theory of explanation methods. Axioms for explanation methods are an interesting theoretical concept to explore their basics and deduce methods. We introduce a new axiom, give its intuition and show it contradicts another existing axiom. Additionally, we point out theoretical difficulties of existing gradient-based and some occlusion-based explanation methods in natural language processing. We provide an extensive argument why evaluation of explanation methods is difficult. We compare OLM to other explanation methods and underline its uniqueness experimentally. Finally, we investigate corner cases of OLM and discuss its validity and possible improvements.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 61 of its 116 references that are in the map. First posted on arXiv 28 Jan 2021.
Builds on
61 of 116 references.
References · 61
- 2017Attention Is All You Needcited by 60,361
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2014Adam: A Method for Stochastic Optimizationcited by 37,240
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
- 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,070
- 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
- 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
- 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
- 2017Axiomatic Attribution for Deep Networkscited by 3,384
- 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
- 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
38 more
- 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,371
- 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
- 2013Visualizing and Understanding Convolutional Networkscited by 2,290
- 2018Deep contextualized word representationscited by 2,290
- 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
- 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
- 2016Enriching Word Vectors with Subword Informationcited by 1,583
- 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
- 2017Learning Important Features Through Propagating Activation Differencescited by 1,458
- 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,416
- 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,319
- 2017Explanation in Artificial Intelligence: Insights from the Social Sciencescited by 1,206
- 2018Sanity Checks for Saliency Mapscited by 1,156
- 2014Convolutional Neural Networks for Sentence Classificationcited by 1,127
- 2017Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPRcited by 1,089
- 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
- 2018A Survey Of Methods For Explaining Black Box Modelscited by 973
- 2018Neural Network Acceptability Judgmentscited by 958
- 2019Adversarial Examples Are Not Bugs, They Are Featurescited by 953
- 2019Attention is not Explanationcited by 929
- 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 876
- 2015Deep Learning and the Information Bottleneck Principlecited by 802
- 2016The Mythos of Model Interpretabilitycited by 721
- 2019Attention is not not Explanationcited by 636
- 2017Interpretable Explanations of Black Boxes by Meaningful Perturbationcited by 612
- 2017Opening the Black Box of Deep Neural Networks via Informationcited by 503
- 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 383
- 2016Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependenciescited by 354
- 2017Methods for Interpreting and Understanding Deep Neural Networkscited by 345
- 2016European Union regulations on algorithmic decision-making and a "right to explanation"cited by 344
- 2017The (Un)reliability of saliency methodscited by 333
- 2017The Expressive Power of Neural Networks: A View from the Widthcited by 320
- 2014The Loss Surfaces of Multilayer Networkscited by 280
- 2018Generalisation in humans and deep neural networkscited by 237
- 2015Visualizing and Understanding Neural Models in NLPcited by 200
- 2019Full-Gradient Representation for Neural Network Visualizationcited by 191
- 2017Visualizing Deep Neural Network Decisions: Prediction Difference Analysiscited by 166
- 2017Interpretable Convolutional Neural Networkscited by 162
50 most cited of 61 All 61 (JSON)
Built on by
Citing papers per year: 2022: 1; 2023: 0; 2024: 0; 2025: 0; 2026: 1.
2 papers.