# An Empirical Study on Explanations in Out-of-Domain Settings

arXiv: https://arxiv.org/abs/2203.00056 · HTML: https://mlsanity.com/paper/2203.00056/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 41 references that are in the map.
- First posted on arXiv 28 Feb 2022.

## Builds on (most cited first, 30 of 31)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1704.02685/) (2017)
- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [Attention is not Explanation](https://mlsanity.com/paper/1902.10186/) (2019)
- [Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?](https://mlsanity.com/paper/2004.03685/) (2020)
- [ERASER: A Benchmark to Evaluate Rationalized NLP Models](https://mlsanity.com/paper/1911.03429/) (2019)
- [e-SNLI: Natural Language Inference with Natural Language Explanations](https://mlsanity.com/paper/1812.01193/) (2018)
- [Is Attention Interpretable?](https://mlsanity.com/paper/1906.03731/) (2019)
- [Explain Yourself! Leveraging Language Models for Commonsense Reasoning](https://mlsanity.com/paper/1906.02361/) (2019)
- [Calibration of Pre-trained Transformers](https://mlsanity.com/paper/2003.07892/) (2020)
- [Pretrained Transformers Improve Out-of-Distribution Robustness](https://mlsanity.com/paper/2004.06100/) (2020)
- [Rationalizing Neural Predictions](https://mlsanity.com/paper/1606.04155/) (2016)
- [Neural Legal Judgment Prediction in English](https://mlsanity.com/paper/1906.02059/) (2019)
- [Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyond](https://mlsanity.com/paper/2109.00725/) (2021)
- [Measuring Association Between Labels and Free-Text Rationales](https://mlsanity.com/paper/2010.12762/) (2020)
- [A Diagnostic Study of Explainability Techniques for Text Classification](https://mlsanity.com/paper/2009.13295/) (2020)
- [Debugging Tests for Model Explanations](https://mlsanity.com/paper/2011.05429/) (2020)
- [Interpretable Neural Predictions with Differentiable Binary Variables](https://mlsanity.com/paper/1905.08160/) (2019)
- [Learning to Faithfully Rationalize by Construction](https://mlsanity.com/paper/2005.00115/) (2020)
- [DeClarE: Debunking Fake News and False Claims using Evidence-Aware Deep Learning](https://mlsanity.com/paper/1809.06416/) (2018)
- [NILE : Natural Language Inference with Faithful Natural Language Explanations](https://mlsanity.com/paper/2005.12116/) (2020)
- [Contextualizing Hate Speech Classifiers with Post-hoc Explanation](https://mlsanity.com/paper/2005.02439/) (2020)
- [Paragraph-level Rationale Extraction through Regularization: A case study on European Court of Human Rights Cases](https://mlsanity.com/paper/2103.13084/) (2021)
- [Investigating the influence of noise and distractors on the interpretation of neural networks](https://mlsanity.com/paper/1611.07270/) (2016)
- [Evaluating and Characterizing Human Rationales](https://mlsanity.com/paper/2010.04736/) (2020)
- [Evaluating the Faithfulness of Importance Measures in NLP by Recursively Masking Allegedly Important Tokens and Retraining](https://mlsanity.com/paper/2110.08412/) (2021)
- [SPECTRA: Sparse Structured Text Rationalization](https://mlsanity.com/paper/2109.04552/) (2021)

## Built on by (most cited first, 21 of 21)

- [CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Task](https://mlsanity.com/paper/2209.06243/) (2022)
- [Inseq: An Interpretability Toolkit for Sequence Generation Models](https://mlsanity.com/paper/2302.13942/) (2023)
- [Model Internals-based Answer Attribution for Trustworthy Retrieval-Augmented Generation](https://mlsanity.com/paper/2406.13663/) (2024)
- [CATfOOD: Counterfactual Augmented Training for Improving Out-of-Domain Performance and Calibration](https://mlsanity.com/paper/2309.07822/) (2023)
- [Investigating Hallucinations in Pruned Large Language Models for Abstractive Summarization](https://mlsanity.com/paper/2311.09335/) (2023)
- [On the Impact of Temporal Concept Drift on Model Explanations](https://mlsanity.com/paper/2210.09197/) (2022)
- [Incorporating Attribution Importance for Improving Faithfulness Metrics](https://mlsanity.com/paper/2305.10496/) (2023)
- [Normalized AOPC: Fixing Misleading Faithfulness Metrics for Feature Attribution Explainability](https://mlsanity.com/paper/2408.08137/) (2024)
- [Calibration Meets Explanation: A Simple and Effective Approach for Model Confidence Estimates](https://mlsanity.com/paper/2211.03041/) (2022)
- [ER-Test: Evaluating Explanation Regularization Methods for Language Models](https://mlsanity.com/paper/2205.12542/) (2022)
- [Comparing Explanation Faithfulness between Multilingual and Monolingual Fine-tuned Language Models](https://mlsanity.com/paper/2403.12809/) (2024)
- [Towards Faithful Explanations for Text Classification with Robustness Improvement and Explanation Guided Training](https://mlsanity.com/paper/2312.17591/) (2023)
- [Noiser: Bounded Input Perturbations for Attributing Large Language Models](https://mlsanity.com/paper/2504.02911/) (2025)
- [Dynamically Refined Regularization for Improving Cross-corpora Hate Speech Detection](https://mlsanity.com/paper/2203.12536/) (2022)
- [Gender Bias in Explainability: Investigating Performance Disparity in Post-hoc Methods](https://mlsanity.com/paper/2505.01198/) (2025)
- [Position: Don't Just "Fix it in Post": A Science of AI Must Study Training Dynamics](https://mlsanity.com/paper/2606.06533/) (2026)
- [Faithfulness and the Notion of Adversarial Sensitivity in NLP Explanations](https://mlsanity.com/paper/2409.17774/) (2024)
- [Function Composition in Trustworthy Machine Learning: Implementation Choices, Insights, and Questions](https://mlsanity.com/paper/2302.09190/) (2023)
- [Estimating Semantic Similarity between In-Domain and Out-of-Domain Samples](https://mlsanity.com/paper/2306.01206/) (2023)
- [Domain Classification-based Source-specific Term Penalization for Domain Adaptation in Hate-speech Detection](https://mlsanity.com/paper/2209.08681/) (2022)
- [Reconstruct Before Summarize: An Efficient Two-Step Framework for Condensing and Summarizing Meeting Transcripts](https://mlsanity.com/paper/2305.07988/) (2023)

## Abstract (arXiv)

Recent work in Natural Language Processing has focused on developing approaches that extract faithful explanations, either via identifying the most important tokens in the input (i.e. post-hoc explanations) or by designing inherently faithful models that first select the most important tokens and then use them to predict the correct label (i.e. select-then-predict models). Currently, these approaches are largely evaluated on in-domain settings. Yet, little is known about how post-hoc explanations and inherently faithful models perform in out-of-domain settings. In this paper, we conduct an extensive empirical study that examines: (1) the out-of-domain faithfulness of post-hoc explanations, generated by five feature attribution methods; and (2) the out-of-domain performance of two inherently faithful models over six datasets. Contrary to our expectations, results show that in many cases out-of-domain post-hoc explanation faithfulness measured by sufficiency and comprehensiveness is higher compared to in-domain. We find this misleading and suggest using a random baseline as a yardstick for evaluating post-hoc explanation faithfulness. Our findings also show that select-then predict models demonstrate comparable predictive performance in out-of-domain settings to full-text trained models.