# Explanation-Based Human Debugging of NLP Models: A Survey

arXiv: https://arxiv.org/abs/2104.15135 · HTML: https://mlsanity.com/paper/2104.15135/

## Key facts

- Cited by 53 papers in the map (data through 14 Oct 2026).
- Builds on 54 of its 115 references that are in the map.
- First posted on arXiv 30 Apr 2021.

## Builds on (most cited first, 30 of 54)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Understanding Black-box Predictions via Influence Functions](https://mlsanity.com/paper/1703.04730/) (2017)
- [Explanation in Artificial Intelligence: Insights from the Social Sciences](https://mlsanity.com/paper/1706.07269/) (2017)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [SmoothGrad: removing noise by adding noise](https://mlsanity.com/paper/1706.03825/) (2017)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [Bag of Tricks for Efficient Text Classification](https://mlsanity.com/paper/1607.01759/) (2016)
- [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://mlsanity.com/paper/1902.01007/) (2019)
- [A Primer in BERTology: What we know about how BERT works](https://mlsanity.com/paper/2002.12327/) (2020)
- [The Mythos of Model Interpretability](https://mlsanity.com/paper/1606.03490/) (2016)
- [Annotation Artifacts in Natural Language Inference Data](https://mlsanity.com/paper/1803.02324/) (2018)
- [Adversarial Examples for Evaluating Reading Comprehension Systems](https://mlsanity.com/paper/1707.07328/) (2017)
- [Pre-trained Models for Natural Language Processing: A Survey](https://mlsanity.com/paper/2003.08271/) (2020)
- [e-SNLI: Natural Language Inference with Natural Language Explanations](https://mlsanity.com/paper/1812.01193/) (2018)
- [Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting](https://mlsanity.com/paper/1901.09451/) (2019)
- [Visual7W: Grounded Question Answering in Images](https://mlsanity.com/paper/1511.03416/) (2015)
- [Adversarial Example Generation with Syntactically Controlled Paraphrase Networks](https://mlsanity.com/paper/1804.06059/) (2018)
- [Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Making](https://mlsanity.com/paper/2001.02114/) (2020)
- [Manipulating and Measuring Model Interpretability](https://mlsanity.com/paper/1802.07810/) (2018)
- [Explainable Machine Learning in Deployment](https://mlsanity.com/paper/1909.06342/) (2019)
- [A Survey of the State of Explainable AI for Natural Language Processing](https://mlsanity.com/paper/2010.00711/) (2020)
- [Informed Machine Learning -- A Taxonomy and Survey of Integrating Knowledge into Learning Systems](https://mlsanity.com/paper/1903.12394/) (2019)
- [Formalizing Trust in Artificial Intelligence: Prerequisites, Causes and Goals of Human Trust in AI](https://mlsanity.com/paper/2010.07487/) (2020)
- [One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques](https://mlsanity.com/paper/1909.03012/) (2019)
- [On Human Predictions with Explanations and Predictions of Machine Learning Models: A Case Study on Deception Detection](https://mlsanity.com/paper/1811.07901/) (2018)
- [Reducing Gender Bias in Abusive Language Detection](https://mlsanity.com/paper/1808.07231/) (2018)

## Built on by (most cited first, 30 of 53)

- [Graph of Thoughts: Solving Elaborate Problems with Large Language Models](https://mlsanity.com/paper/2308.09687/) (2023)
- [Can language models learn from explanations in context?](https://mlsanity.com/paper/2204.02329/) (2022)
- [Towards a Science of Human-AI Decision Making: A Survey of Empirical Studies](https://mlsanity.com/paper/2112.11471/) (2021)
- [GLUE-X: Evaluating Natural Language Understanding Models from an Out-of-distribution Generalization Perspective](https://mlsanity.com/paper/2211.08073/) (2022)
- [Leveraging Explanations in Interactive Machine Learning: An Overview](https://mlsanity.com/paper/2207.14526/) (2022)
- [Concept-level Debugging of Part-Prototype Networks](https://mlsanity.com/paper/2205.15769/) (2022)
- [Disentanglement via Latent Quantization](https://mlsanity.com/paper/2305.18378/) (2023)
- [On the Explainability of Natural Language Processing Deep Models](https://mlsanity.com/paper/2210.06929/) (2022)
- [Tell me why! Explanations support learning relational and causal structure](https://mlsanity.com/paper/2112.03753/) (2021)
- [Going Beyond XAI: A Systematic Survey for Explanation-Guided Learning](https://mlsanity.com/paper/2212.03954/) (2022)
- [ConvXAI: Delivering Heterogeneous AI Explanations via Conversations to Support Human-AI Scientific Writing](https://mlsanity.com/paper/2305.09770/) (2023)
- [A Rationale-Centric Framework for Human-in-the-loop Machine Learning](https://mlsanity.com/paper/2203.12918/) (2022)
- [What to Learn, and How: Toward Effective Learning from Rationales](https://mlsanity.com/paper/2112.00071/) (2021)
- [On Behalf of the Stakeholders: Trends in NLP Model Interpretability in the Era of LLMs](https://mlsanity.com/paper/2407.19200/) (2024)
- [Properties and Challenges of LLM-Generated Explanations](https://mlsanity.com/paper/2402.10532/) (2024)
- [A Survey of Adversarial Defences and Robustness in NLP](https://mlsanity.com/paper/2203.06414/) (2022)
- [Challenges in Applying Explainability Methods to Improve the Fairness of NLP Models](https://mlsanity.com/paper/2206.03945/) (2022)
- [Interpreting Deep Learning Models in Natural Language Processing: A Review](https://mlsanity.com/paper/2110.10470/) (2021)
- [Jointly Attacking Graph Neural Network and its Explanations](https://mlsanity.com/paper/2108.03388/) (2021)
- [Towards Explainable Evaluation Metrics for Natural Language Generation](https://mlsanity.com/paper/2203.11131/) (2022)
- [A survey on improving NLP models with human explanations](https://mlsanity.com/paper/2204.08892/) (2022)
- [Interpretation of Time-Series Deep Models: A Survey](https://mlsanity.com/paper/2305.14582/) (2023)
- [Towards Explainable Evaluation Metrics for Machine Translation](https://mlsanity.com/paper/2306.13041/) (2023)
- [Helpful or Harmful Data? Fine-tuning-free Shapley Attribution for Explaining Language Model Predictions](https://mlsanity.com/paper/2406.04606/) (2024)
- [Contestable AI needs Computational Argumentation](https://mlsanity.com/paper/2405.10729/) (2024)
- [QualEval: Qualitative Evaluation for Model Improvement](https://mlsanity.com/paper/2311.02807/) (2023)
- [ALMANACS: A Simulatability Benchmark for Language Model Explainability](https://mlsanity.com/paper/2312.12747/) (2023)
- [Are Your Explanations Reliable? Investigating the Stability of LIME in Explaining Text Classifiers by Marrying XAI and Adversarial Attack](https://mlsanity.com/paper/2305.12351/) (2023)
- [Human-Centric Research for NLP: Towards a Definition and Guiding Questions](https://mlsanity.com/paper/2207.04447/) (2022)
- [Human-centered NLP Fact-checking: Co-Designing with Fact-checkers using Matchmaking for AI](https://mlsanity.com/paper/2308.07213/) (2023)

## Abstract (arXiv)

Debugging a machine learning model is hard since the bug usually involves the training data and the learning process. This becomes even harder for an opaque deep learning model if we have no clue about how the model actually works. In this survey, we review papers that exploit explanations to enable humans to give feedback and debug NLP models. We call this problem explanation-based human debugging (EBHD). In particular, we categorize and discuss existing work along three dimensions of EBHD (the bug context, the workflow, and the experimental setting), compile findings on how EBHD components affect the feedback providers, and highlight open problems that could be future research directions.