# Explaining Predictions of Non-Linear Classifiers in NLP

arXiv: https://arxiv.org/abs/1606.07298 · HTML: https://mlsanity.com/paper/1606.07298/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Jun 2016.

## Built on by (most cited first, 30 of 37)

- [CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Task](https://mlsanity.com/paper/2209.06243/) (2022)
- [Towards Faithful Model Explanation in NLP: A Survey](https://mlsanity.com/paper/2209.11326/) (2022)
- [On Guaranteed Optimal Robust Explanations for NLP Models](https://mlsanity.com/paper/2105.03640/) (2021)
- [Explanation-Based Human Debugging of NLP Models: A Survey](https://mlsanity.com/paper/2104.15135/) (2021)
- [Walk the Talk? Measuring the Faithfulness of Large Language Model Explanations](https://mlsanity.com/paper/2504.14150/) (2025)
- [Evaluating Saliency Methods for Neural Language Models](https://mlsanity.com/paper/2104.05824/) (2021)
- [Improving the Faithfulness of Attention-based Explanations with Task-specific Information for Text Classification](https://mlsanity.com/paper/2105.02657/) (2021)
- [Mechanistic?](https://mlsanity.com/paper/2410.09087/) (2024)
- [Do Transformer Models Show Similar Attention Patterns to Task-Specific Human Gaze?](https://mlsanity.com/paper/2205.10226/) (2022)
- [Interpreting Deep Learning Models in Natural Language Processing: A Review](https://mlsanity.com/paper/2110.10470/) (2021)
- [On the Lack of Robust Interpretability of Neural Text Classifiers](https://mlsanity.com/paper/2106.04631/) (2021)
- [Enjoy the Salience: Towards Better Transformer-based Faithful Explanations with Word Salience](https://mlsanity.com/paper/2108.13759/) (2021)
- [Towards Explainable Evaluation Metrics for Natural Language Generation](https://mlsanity.com/paper/2203.11131/) (2022)
- [Building a Llama2-finetuned LLM for Odia Language Utilizing Domain Knowledge Instruction Set](https://mlsanity.com/paper/2312.12624/) (2023)
- [Thermostat: A Large Collection of NLP Model Explanations and Analysis Tools](https://mlsanity.com/paper/2108.13961/) (2021)
- [A Close Look at Decomposition-based XAI-Methods for Transformer Language Models](https://mlsanity.com/paper/2502.15886/) (2025)
- [Flexible Instance-Specific Rationalization of NLP Models](https://mlsanity.com/paper/2104.08219/) (2021)
- [Interpretation of Black Box NLP Models: A Survey](https://mlsanity.com/paper/2203.17081/) (2022)
- [A Unified Study of Machine Learning Explanation Evaluation Metrics](https://mlsanity.com/paper/2203.14265/) (2022)
- [XPROAX-Local explanations for text classification with progressive neighborhood approximation](https://mlsanity.com/paper/2109.15004/) (2021)
- [A New Perspective on Evaluation Methods for Explainable Artificial Intelligence (XAI)](https://mlsanity.com/paper/2307.14246/) (2023)
- [Rationale production to support clinical decision-making](https://mlsanity.com/paper/2111.07611/) (2021)
- [More Than Words: Towards Better Quality Interpretations of Text Classifiers](https://mlsanity.com/paper/2112.12444/) (2021)
- [Understanding Text Classification Data and Models Using Aggregated Input Salience](https://mlsanity.com/paper/2211.05485/) (2022)
- [PROMINET: Prototype-based Multi-View Network for Interpretable Email Response Prediction](https://mlsanity.com/paper/2310.16753/) (2023)
- [An Evaluation of Explanation Methods for Black-Box Detectors of Machine-Generated Text](https://mlsanity.com/paper/2408.14252/) (2024)
- [Argumentative Explanations for Pattern-Based Text Classifiers](https://mlsanity.com/paper/2205.10932/) (2022)
- [Explaining text classifiers through progressive neighborhood approximation with realistic samples](https://mlsanity.com/paper/2302.07733/) (2023)
- [Robust Infidelity: When Faithfulness Measures on Masked Language Models Are Misleading](https://mlsanity.com/paper/2308.06795/) (2023)
- [Explaining the Decisions of Deep Policy Networks for Robotic Manipulations](https://mlsanity.com/paper/2310.19432/) (2023)

## Abstract (arXiv)

Layer-wise relevance propagation (LRP) is a recently proposed technique for explaining predictions of complex non-linear classifiers in terms of input variables. In this paper, we apply LRP for the first time to natural language processing (NLP). More precisely, we use it to explain the predictions of a convolutional neural network (CNN) trained on a topic categorization task. Our analysis highlights which words are relevant for a specific prediction of the CNN. We compare our technique to standard sensitivity analysis, both qualitatively and quantitatively, using a "word deleting" perturbation experiment, a PCA analysis, and various visualizations. All experiments validate the suitability of LRP for explaining the CNN predictions, which is also in line with results reported in recent image classification studies.