# Local Interpretations for Explainable Natural Language Processing: A Survey

arXiv: https://arxiv.org/abs/2103.11072 · HTML: https://mlsanity.com/paper/2103.11072/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 126 of its 206 references that are in the map.
- First posted on arXiv 20 Mar 2021.

## Builds on (most cited first, 30 of 126)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1704.02685/) (2017)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [What Does BERT Look At? An Analysis of BERT's Attention](https://mlsanity.com/paper/1906.04341/) (2019)
- [Striving for Simplicity: The All Convolutional Net](https://mlsanity.com/paper/1412.6806/) (2014)
- [A Survey Of Methods For Explaining Black Box Models](https://mlsanity.com/paper/1802.01933/) (2018)
- [VL-BERT: Pre-training of Generic Visual-Linguistic Representations](https://mlsanity.com/paper/1908.08530/) (2019)
- [Attention is not Explanation](https://mlsanity.com/paper/1902.10186/) (2019)
- [SPICE: Semantic Propositional Image Caption Evaluation](https://mlsanity.com/paper/1607.08822/) (2016)
- [The Mythos of Model Interpretability](https://mlsanity.com/paper/1606.03490/) (2016)

## Built on by (most cited first, 30 of 31)

- [Explainability for Large Language Models: A Survey](https://mlsanity.com/paper/2309.01029/) (2023)
- [SaySelf: Teaching LLMs to Express Confidence with Self-Reflective Rationales](https://mlsanity.com/paper/2405.20974/) (2024)
- [UNIREX: A Unified Learning Framework for Language Model Rationale Extraction](https://mlsanity.com/paper/2112.08802/) (2021)
- [On Behalf of the Stakeholders: Trends in NLP Model Interpretability in the Era of LLMs](https://mlsanity.com/paper/2407.19200/) (2024)
- [Ethics Sheet for Automatic Emotion Recognition and Sentiment Analysis](https://mlsanity.com/paper/2109.08256/) (2021)
- [Rationalization for Explainable NLP: A Survey](https://mlsanity.com/paper/2301.08912/) (2023)
- [Multimodal Data Integration for Oncology in the Era of Deep Neural Networks: A Review](https://mlsanity.com/paper/2303.06471/) (2023)
- [Necessity and Sufficiency for Explaining Text Classifiers: A Case Study in Hate Speech Detection](https://mlsanity.com/paper/2205.03302/) (2022)
- [F-Fidelity: A Robust Framework for Faithfulness Evaluation of Explainable AI](https://mlsanity.com/paper/2410.02970/) (2024)
- [A Comprehensive Survey on Self-Interpretable Neural Networks](https://mlsanity.com/paper/2501.15638/) (2025)
- [Doc-GCN: Heterogeneous Graph Convolutional Networks for Document Layout Analysis](https://mlsanity.com/paper/2208.10970/) (2022)
- [Exploring Faithful Rationale for Multi-hop Fact Verification via Salience-Aware Graph Learning](https://mlsanity.com/paper/2212.01060/) (2022)
- [ER-Test: Evaluating Explanation Regularization Methods for Language Models](https://mlsanity.com/paper/2205.12542/) (2022)
- [Penzai + Treescope: A Toolkit for Interpreting, Visualizing, and Editing Models As Data](https://mlsanity.com/paper/2408.00211/) (2024)
- [Explainable Sentiment Analysis with DeepSeek-R1: Performance, Efficiency, and Few-Shot Learning](https://mlsanity.com/paper/2503.11655/) (2025)
- [Understanding Attention for Vision-and-Language Tasks](https://mlsanity.com/paper/2208.08104/) (2022)
- [ExClaim: Explainable Neural Claim Verification Using Rationalization](https://mlsanity.com/paper/2301.08914/) (2023)
- [REFER: An End-to-end Rationale Extraction Framework for Explanation Regularization](https://mlsanity.com/paper/2310.14418/) (2023)
- [SIDU-TXT: An XAI Algorithm for NLP with a Holistic Assessment Approach](https://mlsanity.com/paper/2402.03043/) (2024)
- [Tell Me Why: Explainable Public Health Fact-Checking with Large Language Models](https://mlsanity.com/paper/2405.09454/) (2024)
- [From Fragment to One Piece: A Survey on AI-Driven Graphic Design](https://mlsanity.com/paper/2503.18641/) (2025)
- [Multi-Domain Explainability of Preferences](https://mlsanity.com/paper/2505.20088/) (2025)
- [EMMM, Explain Me My Model! Explainable Machine Generated Text Detection in Dialogues](https://mlsanity.com/paper/2508.18715/) (2025)
- [LIBERTy: A Causal Framework for Benchmarking Concept-Based Explanations of LLMs with Structural Counterfactuals](https://mlsanity.com/paper/2601.10700/) (2026)
- [DeepCodeProbe: Towards Understanding What Models Trained on Code Learn](https://mlsanity.com/paper/2407.08890/) (2024)
- [Make Interval Bound Propagation great again](https://mlsanity.com/paper/2410.03373/) (2024)
- [FarExStance: Explainable Stance Detection for Farsi](https://mlsanity.com/paper/2412.14008/) (2024)
- [Reconnoitering the class distinguishing abilities of the features, to know them better](https://mlsanity.com/paper/2211.12771/) (2022)
- [The Dynamic Gist-Based Memory Model (DGMM): A Memory-Centric Architecture for Artificial Intelligence](https://mlsanity.com/paper/2605.02106/) (2026)
- [MIL-BERT: Classification of Arbitrarily Large Text with Performance and Explanatory Guarantees](https://mlsanity.com/paper/2608.20636/) (2026)

## Abstract (arXiv)

As the use of deep learning techniques has grown across various fields over the past decade, complaints about the opaqueness of the black-box models have increased, resulting in an increased focus on transparency in deep learning models. This work investigates various methods to improve the interpretability of deep neural networks for Natural Language Processing (NLP) tasks, including machine translation and sentiment analysis. We provide a comprehensive discussion on the definition of the term interpretability and its various aspects at the beginning of this work. The methods collected and summarised in this survey are only associated with local interpretation and are specifically divided into three categories: 1) interpreting the model's predictions through related input features; 2) interpreting through natural language explanation; 3) probing the hidden states of models and word representations.