# Connecting Attributions and QA Model Behavior on Realistic Counterfactuals

arXiv: https://arxiv.org/abs/2104.04515 · HTML: https://mlsanity.com/paper/2104.04515/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 48 of its 64 references that are in the map.
- First posted on arXiv 9 Apr 2021.

## Builds on (most cited first, 30 of 48)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples](https://mlsanity.com/paper/1802.00420/) (2018)
- [Explanation in Artificial Intelligence: Insights from the Social Sciences](https://mlsanity.com/paper/1706.07269/) (2017)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [Universal Adversarial Triggers for Attacking and Analyzing NLP](https://mlsanity.com/paper/1908.07125/) (2019)
- [Network Dissection: Quantifying Interpretability of Deep Visual Representations](https://mlsanity.com/paper/1704.05796/) (2017)
- [Designing and Interpreting Probes with Control Tasks](https://mlsanity.com/paper/1909.03368/) (2019)
- [Attention is not not Explanation](https://mlsanity.com/paper/1908.04626/) (2019)
- [Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?](https://mlsanity.com/paper/2004.03685/) (2020)
- [Adversarial Examples for Evaluating Reading Comprehension Systems](https://mlsanity.com/paper/1707.07328/) (2017)
- [e-SNLI: Natural Language Inference with Natural Language Explanations](https://mlsanity.com/paper/1812.01193/) (2018)
- [What you can cram into a single vector: Probing sentence embeddings for linguistic properties](https://mlsanity.com/paper/1805.01070/) (2018)
- [Learning the Difference that Makes a Difference with Counterfactually-Augmented Data](https://mlsanity.com/paper/1909.12434/) (2019)
- [What do you learn from context? Probing for sentence structure in contextualized word representations](https://mlsanity.com/paper/1905.06316/) (2019)
- [Is Attention Interpretable?](https://mlsanity.com/paper/1906.03731/) (2019)
- [Rationalizing Neural Predictions](https://mlsanity.com/paper/1606.04155/) (2016)
- [Selective Question Answering under Domain Shift](https://mlsanity.com/paper/2006.09462/) (2020)
- [Information-Theoretic Probing with Minimum Description Length](https://mlsanity.com/paper/2003.12298/) (2020)
- [Programmatically Interpretable Reinforcement Learning](https://mlsanity.com/paper/1804.02477/) (2018)
- [Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?](https://mlsanity.com/paper/2005.01831/) (2020)
- [Compositional Explanations of Neurons](https://mlsanity.com/paper/2006.14032/) (2020)
- [Measuring Association Between Labels and Free-Text Rationales](https://mlsanity.com/paper/2010.12762/) (2020)

## Built on by (most cited first, 20 of 20)

- [The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning](https://mlsanity.com/paper/2205.03401/) (2022)
- [Towards Faithful Model Explanation in NLP: A Survey](https://mlsanity.com/paper/2209.11326/) (2022)
- [Faith-Shap: The Faithful Shapley Interaction Index](https://mlsanity.com/paper/2203.00870/) (2022)
- [Retrieval-guided Counterfactual Generation for QA](https://mlsanity.com/paper/2110.07596/) (2021)
- [Can Explanations Be Useful for Calibrating Black Box Models?](https://mlsanity.com/paper/2110.07586/) (2021)
- [F-Fidelity: A Robust Framework for Faithfulness Evaluation of Explainable AI](https://mlsanity.com/paper/2410.02970/) (2024)
- [CATfOOD: Counterfactual Augmented Training for Improving Out-of-Domain Performance and Calibration](https://mlsanity.com/paper/2309.07822/) (2023)
- [Incorporating Attribution Importance for Improving Faithfulness Metrics](https://mlsanity.com/paper/2305.10496/) (2023)
- [Machine Reading, Fast and Slow: When Do Models "Understand" Language?](https://mlsanity.com/paper/2209.07430/) (2022)
- [Counterfactual Multihop QA: A Cause-Effect Approach for Reducing Disconnected Reasoning](https://mlsanity.com/paper/2210.07138/) (2022)
- [Assessing Out-of-Domain Language Model Performance from Few Examples](https://mlsanity.com/paper/2210.06725/) (2022)
- [Hallucination Augmented Recitations for Language Models](https://mlsanity.com/paper/2311.07424/) (2023)
- [Single Sequence Prediction over Reasoning Graphs for Multi-hop QA](https://mlsanity.com/paper/2307.00335/) (2023)
- [Evaluating Input Feature Explanations through a Unified Diagnostic Evaluation Framework](https://mlsanity.com/paper/2406.15085/) (2024)
- [Efficient Shapley Values Estimation by Amortization for Text Classification](https://mlsanity.com/paper/2305.19998/) (2023)
- [Exploring Contrast Consistency of Open-Domain Question Answering Systems on Minimally Edited Questions](https://mlsanity.com/paper/2305.14441/) (2023)
- [Explaining Interactions Between Text Spans](https://mlsanity.com/paper/2310.13506/) (2023)
- [Conversational Explanations: Discussing Explainable AI with Non-AI Experts](https://mlsanity.com/paper/2503.16444/) (2025)
- [Cross-Attention is Half Explanation in Speech-to-Text Models](https://mlsanity.com/paper/2509.18010/) (2025)
- [NaturalAdversaries: Can Naturalistic Adversaries Be as Effective as Artificial Adversaries?](https://mlsanity.com/paper/2211.04364/) (2022)

## Abstract (arXiv)

When a model attribution technique highlights a particular part of the input, a user might understand this highlight as making a statement about counterfactuals (Miller, 2019): if that part of the input were to change, the model's prediction might change as well. This paper investigates how well different attribution techniques align with this assumption on realistic counterfactuals in the case of reading comprehension (RC). RC is a particularly challenging test case, as token-level attributions that have been extensively studied in other NLP tasks such as sentiment analysis are less suitable to represent the reasoning that RC models perform. We construct counterfactual sets for three different RC settings, and through heuristics that can connect attribution methods' outputs to high-level model behavior, we can evaluate how useful different attribution methods and even different formats are for understanding counterfactuals. We find that pairwise attributions are better suited to RC than token-level attributions across these different RC settings, with our best performance coming from a modification that we propose to an existing pairwise attribution method.