# "Will You Find These Shortcuts?" A Protocol for Evaluating the Faithfulness of Input Salience Methods for Text Classification

arXiv: https://arxiv.org/abs/2111.07367 · HTML: https://mlsanity.com/paper/2111.07367/

## Key facts

- Cited by 66 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 67 references that are in the map.
- First posted on arXiv 14 Nov 2021.

## Builds on (most cited first, 30 of 46)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Attention is not Explanation](https://mlsanity.com/paper/1902.10186/) (2019)
- [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://mlsanity.com/paper/1902.01007/) (2019)
- [Annotation Artifacts in Natural Language Inference Data](https://mlsanity.com/paper/1803.02324/) (2018)
- [Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)](https://mlsanity.com/paper/1902.03368/) (2019)
- [Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?](https://mlsanity.com/paper/2004.03685/) (2020)
- [ERASER: A Benchmark to Evaluate Rationalized NLP Models](https://mlsanity.com/paper/1911.03429/) (2019)
- [A Benchmark for Interpretability Methods in Deep Neural Networks](https://mlsanity.com/paper/1806.10758/) (2018)
- [Learning the Difference that Makes a Difference with Counterfactually-Augmented Data](https://mlsanity.com/paper/1909.12434/) (2019)
- [Generating Radiology Reports via Memory-driven Transformer](https://mlsanity.com/paper/2010.16056/) (2020)
- [The (Un)reliability of saliency methods](https://mlsanity.com/paper/1711.00867/) (2017)
- [On the (In)fidelity and Sensitivity for Explanations](https://mlsanity.com/paper/1901.09392/) (2019)
- [Hypothesis Only Baselines in Natural Language Inference](https://mlsanity.com/paper/1805.01042/) (2018)
- [Ex Machina: Personal Attacks Seen at Scale](https://mlsanity.com/paper/1610.08914/) (2016)
- [Visualizing and Understanding Neural Models in NLP](https://mlsanity.com/paper/1506.01066/) (2015)
- [Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasets](https://mlsanity.com/paper/1908.07898/) (2019)
- [A Diagnostic Study of Explainability Techniques for Text Classification](https://mlsanity.com/paper/2009.13295/) (2020)
- [Debugging Tests for Model Explanations](https://mlsanity.com/paper/2011.05429/) (2020)
- [Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions](https://mlsanity.com/paper/2005.06676/) (2020)
- [The Language Interpretability Tool: Extensible, Interactive Visualizations and Analysis for NLP Models](https://mlsanity.com/paper/2008.05122/) (2020)
- [The elephant in the interpretability room: Why use attention as explanation when we have saliency methods?](https://mlsanity.com/paper/2010.05607/) (2020)
- [Explaining NLP Models via Minimal Contrastive Editing (MiCE)](https://mlsanity.com/paper/2012.13985/) (2020)
- [Do Feature Attribution Methods Correctly Attribute Features?](https://mlsanity.com/paper/2104.14403/) (2021)
- [Benchmarking Attribution Methods with Relative Feature Importance](https://mlsanity.com/paper/1907.09701/) (2019)
- [Towards Interpreting and Mitigating Shortcut Learning Behavior of NLU Models](https://mlsanity.com/paper/2103.06922/) (2021)
- [Did the Model Understand the Question?](https://mlsanity.com/paper/1805.05492/) (2018)

## Built on by (most cited first, 30 of 66)

- [Explainability for Large Language Models: A Survey](https://mlsanity.com/paper/2309.01029/) (2023)
- [Protein Design with Guided Discrete Diffusion](https://mlsanity.com/paper/2305.20009/) (2023)
- [Towards Faithful Model Explanation in NLP: A Survey](https://mlsanity.com/paper/2209.11326/) (2022)
- [Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks](https://mlsanity.com/paper/2207.13243/) (2022)
- [Rethinking Interpretability in the Era of Large Language Models](https://mlsanity.com/paper/2402.01761/) (2024)
- [Are self-explanations from Large Language Models faithful?](https://mlsanity.com/paper/2401.07927/) (2024)
- [Can Large Language Models Explain Themselves? A Study of LLM-Generated Self-Explanations](https://mlsanity.com/paper/2310.11207/) (2023)
- [Inseq: An Interpretability Toolkit for Sequence Generation Models](https://mlsanity.com/paper/2302.13942/) (2023)
- [Measuring the Mixing of Contextual Information in the Transformer](https://mlsanity.com/paper/2203.04212/) (2022)
- [Post hoc Explanations may be Ineffective for Detecting Unknown Spurious Correlation](https://mlsanity.com/paper/2212.04629/) (2022)
- [NNsight and NDIF: Democratizing Access to Open-Weight Foundation Model Internals](https://mlsanity.com/paper/2407.14561/) (2024)
- [DecompX: Explaining Transformers Decisions by Propagating Token Decomposition](https://mlsanity.com/paper/2306.02873/) (2023)
- [Model Internals-based Answer Attribution for Trustworthy Retrieval-Augmented Generation](https://mlsanity.com/paper/2406.13663/) (2024)
- [Learning to Scaffold: Optimizing Model Explanations for Teaching](https://mlsanity.com/paper/2204.10810/) (2022)
- [Diagnosing AI Explanation Methods with Folk Concepts of Behavior](https://mlsanity.com/paper/2201.11239/) (2022)
- [Finding Dataset Shortcuts with Grammar Induction](https://mlsanity.com/paper/2210.11560/) (2022)
- [Are Character-level Translations Worth the Wait? Comparing ByT5 and mT5 for Machine Translation](https://mlsanity.com/paper/2302.14220/) (2023)
- [The Inside Story: Towards Better Understanding of Machine Translation Neural Evaluation Metrics](https://mlsanity.com/paper/2305.11806/) (2023)
- [Stability Guarantees for Feature Attributions with Multiplicative Smoothing](https://mlsanity.com/paper/2307.05902/) (2023)
- [Fuse to Forget: Bias Reduction and Selective Memorization through Model Fusion](https://mlsanity.com/paper/2311.07682/) (2023)
- [The Solvability of Interpretability Evaluation Metrics](https://mlsanity.com/paper/2205.08696/) (2022)
- [Computational Experiments Meet Large Language Model Based Agents: A Survey and Perspective](https://mlsanity.com/paper/2402.00262/) (2024)
- [Interpretability Needs a New Paradigm](https://mlsanity.com/paper/2405.05386/) (2024)
- [A Close Look at Decomposition-based XAI-Methods for Transformer Language Models](https://mlsanity.com/paper/2502.15886/) (2025)
- [Transparency at the Source: Evaluating and Interpreting Language Models With Access to the True Distribution](https://mlsanity.com/paper/2310.14840/) (2023)
- [On the Impact of Knowledge Distillation for Model Interpretability](https://mlsanity.com/paper/2305.15734/) (2023)
- [Feature Interactions Reveal Linguistic Structure in Language Models](https://mlsanity.com/paper/2306.12181/) (2023)
- [Randomly Removing 50% of Dimensions in Text Embeddings has Minimal Impact on Retrieval and Classification Tasks](https://mlsanity.com/paper/2508.17744/) (2025)
- [Adversarial Attacks on the Interpretation of Neuron Activation Maximization](https://mlsanity.com/paper/2306.07397/) (2023)
- [Quantifying the Plausibility of Context Reliance in Neural Machine Translation](https://mlsanity.com/paper/2310.01188/) (2023)

## Abstract (arXiv)

Feature attribution a.k.a. input salience methods which assign an importance score to a feature are abundant but may produce surprisingly different results for the same model on the same input. While differences are expected if disparate definitions of importance are assumed, most methods claim to provide faithful attributions and point at the features most relevant for a model's prediction. Existing work on faithfulness evaluation is not conclusive and does not provide a clear answer as to how different methods are to be compared. Focusing on text classification and the model debugging scenario, our main contribution is a protocol for faithfulness evaluation that makes use of partially synthetic data to obtain ground truth for feature importance ranking. Following the protocol, we do an in-depth analysis of four standard salience method classes on a range of datasets and shortcuts for BERT and LSTM models and demonstrate that some of the most popular method configurations provide poor results even for simplest shortcuts. We recommend following the protocol for each new task and model combination to find the best method for identifying shortcuts.