mlsanityData to 7 Oct 2026

Connecting Attributions and QA Model Behavior on Realistic Counterfactuals

arXiv 2104.04515 ↗First posted 9 Apr 2021cs.CLcited by 20

Not on the map. Why?

Abstract (arXiv)

When a model attribution technique highlights a particular part of the input, a user might understand this highlight as making a statement about counterfactuals (Miller, 2019): if that part of the input were to change, the model's prediction might change as well. This paper investigates how well different attribution techniques align with this assumption on realistic counterfactuals in the case of reading comprehension (RC). RC is a particularly challenging test case, as token-level attributions that have been extensively studied in other NLP tasks such as sentiment analysis are less suitable to represent the reasoning that RC models perform. We construct counterfactual sets for three different RC settings, and through heuristics that can connect attribution methods' outputs to high-level model behavior, we can evaluate how useful different attribution methods and even different formats are for understanding counterfactuals. We find that pairwise attributions are better suited to RC than token-level attributions across these different RC settings, with our best performance coming from a modification that we propose to an existing pairwise attribution method.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 48 of its 64 references that are in the map. First posted on arXiv 9 Apr 2021.

Builds on

48 of 64 references.

References · 48

  1. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  2. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  3. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  4. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  5. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  6. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  7. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
  8. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  9. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,400
  10. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,259
  11. 2017Explanation in Artificial Intelligence: Insights from the Social Sciencescited by 1,201
  12. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
36 more
  1. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
  2. 2017Network Dissection: Quantifying Interpretability of Deep Visual Representationscited by 703
  3. 2019Attention is not not Explanationcited by 629
  4. 2019Designing and Interpreting Probes with Control Taskscited by 608
  5. 2020Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?cited by 590
  6. 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 568
  7. 2018e-SNLI: Natural Language Inference with Natural Language Explanationscited by 468
  8. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 452
  9. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 436
  10. 2019Learning the Difference that Makes a Difference with Counterfactually-Augmented Datacited by 428
  11. 2019Is Attention Interpretable?cited by 387
  12. 2016Rationalizing Neural Predictionscited by 281
  13. 2020Information-Theoretic Probing with Minimum Description Lengthcited by 230
  14. 2020Selective Question Answering under Domain Shiftcited by 228
  15. 2018Programmatically Interpretable Reinforcement Learningcited by 192
  16. 2020Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?cited by 189
  17. 2020Compositional Explanations of Neuronscited by 172
  18. 2020Measuring Association Between Labels and Free-Text Rationalescited by 159
  19. 2016Generating Visual Explanationscited by 156
  20. 2020Self-Attention Attribution: Interpreting Information Interactions Inside Transformercited by 124
  21. 2019Compositional Questions Do Not Necessitate Multi-hop Reasoningcited by 87
  22. 2019Counterfactual Story Reasoning and Generationcited by 87
  23. 2020Aligning Faithful Interpretations with their Social Attributioncited by 79
  24. 2020How does this interaction affect me? Interpretable attribution for feature interactionscited by 73
  25. 2018Faithful Multimodal Explanation for Visual Question Answeringcited by 64
  26. 2019Avoiding Reasoning Shortcuts: Adversarial Evaluation, Training, and Model Development for Multi-Hop QAcited by 63
  27. 2020How do Decisions Emerge across Layers in Neural Models? Interpretation with Differentiable Maskingcited by 63
  28. 2019Select, Answer and Explain: Interpretable Multi-hop Reading Comprehension over Multiple Documentscited by 57
  29. 2020Generating Hierarchical Explanations on Text Classification via Feature Interaction Detectioncited by 57
  30. 2018Do Explanations make VQA Models more Predictable to a Human?cited by 51
  31. 2019Understanding Dataset Design Choices for Multi-hop Reasoningcited by 50
  32. 2019Do Human Rationales Improve Machine Explanations?cited by 33
  33. 2019Generating Token-Level Explanations for Natural Language Inferencecited by 28
  34. 2020A Simple Yet Strong Pipeline for HotpotQAcited by 24
  35. 2020Obtaining Faithful Interpretations from Compositional Neural Networkscited by 23
  36. 2020Considering Likelihood in NLP Classification Explanations with Occlusion and Language Modelingcited by 11

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 7; 2023: 7; 2024: 2; 2025: 2.

20 papers.

Sort:

By area:

  1. 2022The Unreliability of Explanations in Few-shot Prompting for Textual Reasoningcited by 169
  2. 2022Towards Faithful Model Explanation in NLP: A Surveycited by 129
  3. 2022Faith-Shap: The Faithful Shapley Interaction Indexcited by 68
  4. 2021Retrieval-guided Counterfactual Generation for QAcited by 28
  5. 2021Can Explanations Be Useful for Calibrating Black Box Models?cited by 20
  6. 2024F-Fidelity: A Robust Framework for Faithfulness Evaluation of Explainable AIcited by 17
  7. 2023CATfOOD: Counterfactual Augmented Training for Improving Out-of-Domain Performance and Calibrationcited by 16
  8. 2023Incorporating Attribution Importance for Improving Faithfulness Metricscited by 14
  9. 2022Machine Reading, Fast and Slow: When Do Models "Understand" Language?cited by 13
  10. 2022Assessing Out-of-Domain Language Model Performance from Few Examplescited by 6
  11. 2022Counterfactual Multihop QA: A Cause-Effect Approach for Reducing Disconnected Reasoningcited by 6
  12. 2023Single Sequence Prediction over Reasoning Graphs for Multi-hop QAcited by 5
8 more
  1. 2023Hallucination Augmented Recitations for Language Modelscited by 5
  2. 2024Evaluating Input Feature Explanations through a Unified Diagnostic Evaluation Frameworkcited by 5
  3. 2023Efficient Shapley Values Estimation by Amortization for Text Classificationcited by 4
  4. 2023Exploring Contrast Consistency of Open-Domain Question Answering Systems on Minimally Edited Questionscited by 3
  5. 2023Explaining Interactions Between Text Spanscited by 1
  6. 2025Conversational Explanations: Discussing Explainable AI with Non-AI Expertscited by 1
  7. 2025Cross-Attention is Half Explanation in Speech-to-Text Modelscited by 1
  8. 2022NaturalAdversaries: Can Naturalistic Adversaries Be as Effective as Artificial Adversaries?cited by 0