# On the Sensitivity and Stability of Model Interpretations in NLP

arXiv: https://arxiv.org/abs/2104.08782 · HTML: https://mlsanity.com/paper/2104.08782/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 47 references that are in the map.
- First posted on arXiv 18 Apr 2021.

## Builds on (most cited first, 30 of 35)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1704.02685/) (2017)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [Attention is not Explanation](https://mlsanity.com/paper/1902.10186/) (2019)
- [Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?](https://mlsanity.com/paper/2004.03685/) (2020)
- [ERASER: A Benchmark to Evaluate Rationalized NLP Models](https://mlsanity.com/paper/1911.03429/) (2019)
- [Interpretation of Neural Networks is Fragile](https://mlsanity.com/paper/1710.10547/) (2017)
- [Generating Natural Language Adversarial Examples](https://mlsanity.com/paper/1804.07998/) (2018)
- [Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints](https://mlsanity.com/paper/1707.09457/) (2017)
- [Is Attention Interpretable?](https://mlsanity.com/paper/1906.03731/) (2019)
- [Deep Biaffine Attention for Neural Dependency Parsing](https://mlsanity.com/paper/1611.01734/) (2016)
- [The (Un)reliability of saliency methods](https://mlsanity.com/paper/1711.00867/) (2017)
- [On the Effectiveness of Interval Bound Propagation for Training Verifiably Robust Models](https://mlsanity.com/paper/1810.12715/) (2018)
- [On the Robustness of Interpretability Methods](https://mlsanity.com/paper/1806.08049/) (2018)
- [On the (In)fidelity and Sensitivity for Explanations](https://mlsanity.com/paper/1901.09392/) (2019)
- [Visualizing and Understanding Neural Models in NLP](https://mlsanity.com/paper/1506.01066/) (2015)
- [Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?](https://mlsanity.com/paper/2005.01831/) (2020)
- [Certified Robustness to Adversarial Word Substitutions](https://mlsanity.com/paper/1909.00986/) (2019)
- [A Diagnostic Study of Explainability Techniques for Text Classification](https://mlsanity.com/paper/2009.13295/) (2020)
- [Pathologies of Neural Models Make Interpretations Difficult](https://mlsanity.com/paper/1804.07781/) (2018)
- [The elephant in the interpretability room: Why use attention as explanation when we have saliency methods?](https://mlsanity.com/paper/2010.05607/) (2020)
- [Evaluating Explanations: How much do explanations from the teacher aid students?](https://mlsanity.com/paper/2012.00893/) (2020)

## Built on by (most cited first, 30 of 30)

- [Towards Faithful Model Explanation in NLP: A Survey](https://mlsanity.com/paper/2209.11326/) (2022)
- [Faithfulness Tests for Natural Language Explanations](https://mlsanity.com/paper/2305.18029/) (2023)
- ["Will You Find These Shortcuts?" A Protocol for Evaluating the Faithfulness of Input Salience Methods for Text Classification](https://mlsanity.com/paper/2111.07367/) (2021)
- [Faithful Explanations of Black-box NLP Models Using LLM-generated Counterfactuals](https://mlsanity.com/paper/2310.00603/) (2023)
- [Don't Lie to Me! Robust and Efficient Explainability with Verified Perturbation Analysis](https://mlsanity.com/paper/2202.07728/) (2022)
- [ferret: a Framework for Benchmarking Explainers on Transformers](https://mlsanity.com/paper/2208.01575/) (2022)
- [SEAT: Stable and Explainable Attention](https://mlsanity.com/paper/2211.13290/) (2022)
- [SATO: Stable Text-to-Motion Framework](https://mlsanity.com/paper/2405.01461/) (2024)
- [Generic Temporal Reasoning with Differential Analysis and Explanation](https://mlsanity.com/paper/2212.10467/) (2022)
- [F-Fidelity: A Robust Framework for Faithfulness Evaluation of Explainable AI](https://mlsanity.com/paper/2410.02970/) (2024)
- [A Fine-grained Interpretability Evaluation Benchmark for Neural NLP](https://mlsanity.com/paper/2205.11097/) (2022)
- [Incorporating Attribution Importance for Improving Faithfulness Metrics](https://mlsanity.com/paper/2305.10496/) (2023)
- [Prompt-Agnostic Adversarial Perturbation for Customized Diffusion Models](https://mlsanity.com/paper/2408.10571/) (2024)
- [Improving Interpretation Faithfulness for Vision Transformers](https://mlsanity.com/paper/2311.17983/) (2023)
- [Evaluating Human Alignment and Model Faithfulness of LLM Rationale](https://mlsanity.com/paper/2407.00219/) (2024)
- [ALMANACS: A Simulatability Benchmark for Language Model Explainability](https://mlsanity.com/paper/2312.12747/) (2023)
- [Normalized AOPC: Fixing Misleading Faithfulness Metrics for Feature Attribution Explainability](https://mlsanity.com/paper/2408.08137/) (2024)
- [On the Impact of Knowledge Distillation for Model Interpretability](https://mlsanity.com/paper/2305.15734/) (2023)
- [Human Attention-Guided Explainable Artificial Intelligence for Computer Vision Models](https://mlsanity.com/paper/2305.03601/) (2023)
- [DRIVE: Dependable Robust Interpretable Visionary Ensemble Framework in Autonomous Driving](https://mlsanity.com/paper/2409.10330/) (2024)
- [An Interpretability Evaluation Benchmark for Pre-trained Language Models](https://mlsanity.com/paper/2207.13948/) (2022)
- [Efficient Shapley Values Estimation by Amortization for Text Classification](https://mlsanity.com/paper/2305.19998/) (2023)
- [Faithful Interpretation for Graph Neural Networks](https://mlsanity.com/paper/2410.06950/) (2024)
- [CERET: Cost-Effective Extrinsic Refinement for Text Generation](https://mlsanity.com/paper/2406.05588/) (2024)
- [Explainable AI the Latest Advancements and New Trends](https://mlsanity.com/paper/2505.07005/) (2025)
- [Influence based explainability of brain tumors segmentation in multimodal Magnetic Resonance Imaging](https://mlsanity.com/paper/2405.12222/) (2024)
- [When Machine Learning Gets Personal: Evaluating Prediction and Explanation](https://mlsanity.com/paper/2502.02786/) (2025)
- [Multilingual Relative Clause Attachment Ambiguity Resolution in Large Language Models](https://mlsanity.com/paper/2503.02971/) (2025)
- [Application of integrated gradients explainability to sociopsychological semantic markers](https://mlsanity.com/paper/2503.04989/) (2025)
- [Faithfulness Evaluation for Decoder-only LLM Attributions with Controlled Retained Information](https://mlsanity.com/paper/2601.03089/) (2026)

## Abstract (arXiv)

Recent years have witnessed the emergence of a variety of post-hoc interpretations that aim to uncover how natural language processing (NLP) models make predictions. Despite the surge of new interpretation methods, it remains an open problem how to define and quantitatively measure the faithfulness of interpretations, i.e., to what extent interpretations reflect the reasoning process by a model. We propose two new criteria, sensitivity and stability, that provide complementary notions of faithfulness to the existed removal-based criteria. Our results show that the conclusion for how faithful interpretations are could vary substantially based on different notions. Motivated by the desiderata of sensitivity and stability, we introduce a new class of interpretation methods that adopt techniques from adversarial robustness. Empirical results show that our proposed methods are effective under the new criteria and overcome limitations of gradient-based methods on removal-based criteria. Besides text classification, we also apply interpretation methods and metrics to dependency parsing. Our results shed light on understanding the diverse set of interpretations.