mlsanityData to 7 Oct 2026

Explaining the Road Not Taken

arXiv 2103.14973 ↗First posted 27 Mar 2021cs.CL cs.AI cs.HCcited by 3

Not on the map. Why?

Abstract (arXiv)

It is unclear if existing interpretations of deep neural network models respond effectively to the needs of users. This paper summarizes the common forms of explanations (such as feature attribution, decision rules, or probes) used in over 200 recent papers about natural language processing (NLP), and compares them against user questions collected in the XAI Question Bank. We found that although users are interested in explanations for the road not taken -- namely, why the model chose one result and not a well-defined, seemly similar legitimate counterpart -- most model interpretations cannot answer these questions.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 42 of its 55 references that are in the map. First posted on arXiv 27 Mar 2021.

Builds on

42 of 55 references.

References · 42

  1. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  2. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  3. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
  4. 2019Explaining Machine Learning Classifiers through Diverse Counterfactual Explanationscited by 612
  5. 2019Designing and Interpreting Probes with Control Taskscited by 608
  6. 2020Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?cited by 590
  7. 2018e-SNLI: Natural Language Inference with Natural Language Explanationscited by 468
  8. 2019Learning the Difference that Makes a Difference with Counterfactually-Augmented Datacited by 428
  9. 2019Explain Yourself! Leveraging Language Models for Commonsense Reasoningcited by 367
  10. 2017Methods for Interpreting and Understanding Deep Neural Networkscited by 342
  11. 2019What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language modelscited by 317
  12. 2016Rationalizing Neural Predictionscited by 281
30 more
  1. 2020Questioning the AI: Informing Design Practices for Explainable AI User Experiencescited by 251
  2. 2020Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?cited by 189
  3. 2018Representer Point Selection for Explaining Deep Neural Networkscited by 187
  4. 2018Explaining Image Classifiers by Counterfactual Generationcited by 144
  5. 2020Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functionscited by 144
  6. 2018Pathologies of Neural Models Make Interpretations Difficultcited by 140
  7. 2020The elephant in the interpretability room: Why use attention as explanation when we have saliency methods?cited by 122
  8. 2019Open Sesame: Getting Inside BERT's Linguistic Knowledgecited by 119
  9. 2020Explaining NLP Models via Minimal Contrastive Editing (MiCE)cited by 105
  10. 2020Vokenization: Improving Language Understanding with Contextualized, Visual-Grounded Supervisioncited by 93
  11. 2021Contrastive Explanations for Model Interpretabilitycited by 75
  12. 2018Seq2Seq-Vis: A Visual Debugging Tool for Sequence-to-Sequence Modelscited by 69
  13. 2019Self-Critical Reasoning for Robust Visual Question Answeringcited by 61
  14. 2018Did the Model Understand the Question?cited by 60
  15. 2020PRover: Proof Generation for Interpretable Reasoning over Rulescited by 53
  16. 2018What can AI do for me: Evaluating Machine Learning Interpretations in Cooperative Playcited by 52
  17. 2019Do Neural Dialog Systems Use the Conversation History Effectively? An Empirical Studycited by 47
  18. 2019Multi-hop Question Answering via Reasoning Chainscited by 42
  19. 2020FIND: Human-in-the-Loop Debugging Deep Text Classifierscited by 40
  20. 2020How Useful Are the Machine-Generated Interpretations to General Users? A Human Evaluation on Guessing the Incorrectly Predicted Labelscited by 39
  21. 2020Are Visual Explanations Useful? A Case Study in Model-in-the-Loop Predictioncited by 36
  22. 2020Learning Variational Word Masks to Improve the Interpretability of Neural Text Classifierscited by 35
  23. 2020ViCE: Visual Counterfactual Explanations for Machine Learning Modelscited by 34
  24. 2019Self-Assembling Modular Networks for Interpretable Multi-Hop Reasoningcited by 28
  25. 2019Investigating Robustness and Interpretability of Link Prediction via Adversarial Modificationscited by 25
  26. 2016Interpreting Neural Networks to Improve Politeness Comprehensioncited by 12
  27. 2020LEAN-LIFE: A Label-Efficient Annotation Framework Towards Learning from Explanationcited by 11
  28. 2018Lightly-supervised Representation Learning with Global Interpretabilitycited by 6
  29. 2019Neural Vector Conceptualization for Word Vector Space Interpretationcited by 3
  30. 2017Unsupervised, Knowledge-Free, and Interpretable Word Sense Disambiguationcited by 2

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 1.

3 papers.

  1. 2023ConvXAI: Delivering Heterogeneous AI Explanations via Conversations to Support Human-AI Scientific Writingcited by 31
  2. 2022Beware the Rationalization Trap! When Language Model Explainability Diverges from our Mental Models of Languagecited by 9
  3. 2022Are Shortest Rationales the Best Explanations for Human Understanding?cited by 8