# Model Explainability in Deep Learning Based Natural Language Processing

arXiv: https://arxiv.org/abs/2106.07410 · HTML: https://mlsanity.com/paper/2106.07410/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 26 references that are in the map.
- First posted on arXiv 14 Jun 2021.

## Builds on (most cited first, 19 of 19)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1704.02685/) (2017)
- [Distribution-Free Predictive Inference For Regression](https://mlsanity.com/paper/1604.04173/) (2016)
- [Methods for Interpreting and Understanding Deep Neural Networks](https://mlsanity.com/paper/1706.07979/) (2017)
- [Model-Agnostic Interpretability of Machine Learning](https://mlsanity.com/paper/1606.05386/) (2016)
- [Neural Language Models as Psycholinguistic Subjects: Representations of Syntactic State](https://mlsanity.com/paper/1903.03260/) (2019)
- [Explaining Recurrent Neural Network Predictions in Sentiment Analysis](https://mlsanity.com/paper/1706.07206/) (2017)
- [iNNvestigate neural networks!](https://mlsanity.com/paper/1808.04260/) (2018)
- [Towards Explainable NLP: A Generative Explanation Framework for Text Classification](https://mlsanity.com/paper/1811.00196/) (2018)
- ["What is Relevant in a Text Document?": An Interpretable Machine Learning Approach](https://mlsanity.com/paper/1612.07843/) (2016)
- [Explainable Recommendation via Multi-Task Learning in Opinionated Text Data](https://mlsanity.com/paper/1806.03568/) (2018)
- [Understanding Convolutional Neural Networks for Text Classification](https://mlsanity.com/paper/1809.08037/) (2018)
- [Explainable Neural Networks based on Additive Index Models](https://mlsanity.com/paper/1806.01933/) (2018)
- [Locally Interpretable Models and Effects based on Supervised Partitioning (LIME-SUP)](https://mlsanity.com/paper/1806.00663/) (2018)
- [SHAP values for Explaining CNN-based Text Classification Models](https://mlsanity.com/paper/2008.11825/) (2020)
- [Self-interpretable Convolutional Neural Networks for Text Classification](https://mlsanity.com/paper/2105.08589/) (2021)
- [Model Interpretation: A Unified Derivative-based Framework for Nonparametric Regression and Supervised Machine Learning](https://mlsanity.com/paper/1808.07216/) (2018)
- [Analyzing and Interpreting Convolutional Neural Networks in NLP](https://mlsanity.com/paper/1810.09312/) (2018)

## Built on by (most cited first, 14 of 14)

- [Explainable Artificial Intelligence: A Survey of Needs, Techniques, Applications, and Future Direction](https://mlsanity.com/paper/2409.00265/) (2024)
- [Interpreting Deep Learning Models in Natural Language Processing: A Review](https://mlsanity.com/paper/2110.10470/) (2021)
- [Interpretable Medical Diagnostics with Structured Data Extraction by Large Language Models](https://mlsanity.com/paper/2306.05052/) (2023)
- [A Survey on Explainable Artificial Intelligence for Cybersecurity](https://mlsanity.com/paper/2303.12942/) (2023)
- [Evaluating and Explaining Large Language Models for Code Using Syntactic Structures](https://mlsanity.com/paper/2308.03873/) (2023)
- [Explainability in Neural Networks for Natural Language Processing Tasks](https://mlsanity.com/paper/2412.18036/) (2024)
- [Explainable Misinformation Detection Across Multiple Social Media Platforms](https://mlsanity.com/paper/2203.11724/) (2022)
- [EAG-RS: A Novel Explainability-guided ROI-Selection Framework for ASD Diagnosis via Inter-regional Relation Learning](https://mlsanity.com/paper/2310.03404/) (2023)
- [Lightweight Safety Classification Using Pruned Language Models](https://mlsanity.com/paper/2412.13435/) (2024)
- [Accelerating the Global Aggregation of Local Explanations](https://mlsanity.com/paper/2312.07991/) (2023)
- [Fake News Detection After LLM Laundering: Measurement and Explanation](https://mlsanity.com/paper/2501.18649/) (2025)
- [Multi-resolution Interpretation and Diagnostics Tool for Natural Language Classifiers](https://mlsanity.com/paper/2303.03542/) (2023)
- [Unlocking Layer-wise Relevance Propagation for Autoencoders](https://mlsanity.com/paper/2303.11734/) (2023)
- [Integrating Natural Language Processing Techniques of Text Mining Into Financial System: Applications and Limitations](https://mlsanity.com/paper/2412.20438/) (2024)

## Abstract (arXiv)

Machine learning (ML) model explainability has received growing attention, especially in the area related to model risk and regulations. In this paper, we reviewed and compared some popular ML model explainability methodologies, especially those related to Natural Language Processing (NLP) models. We then applied one of the NLP explainability methods Layer-wise Relevance Propagation (LRP) to a NLP classification model. We used the LRP method to derive a relevance score for each word in an instance, which is a local explainability. The relevance scores are then aggregated together to achieve global variable importance of the model. Through the case study, we also demonstrated how to apply the local explainability method to false positive and false negative instances to discover the weakness of a NLP model. These analysis can help us to understand NLP models better and reduce the risk due to the black-box nature of NLP models. We also identified some common issues due to the special natures of NLP models and discussed how explainability analysis can act as a control to detect these issues after the model has been trained.