mlsanityData to 7 Oct 2026

"Will You Find These Shortcuts?" A Protocol for Evaluating the Faithfulness of Input Salience Methods for Text Classification

arXiv 2111.07367 ↗First posted 14 Nov 2021cs.CLcited by 66

Not on the map. Why?

Abstract (arXiv)

Feature attribution a.k.a. input salience methods which assign an importance score to a feature are abundant but may produce surprisingly different results for the same model on the same input. While differences are expected if disparate definitions of importance are assumed, most methods claim to provide faithful attributions and point at the features most relevant for a model's prediction. Existing work on faithfulness evaluation is not conclusive and does not provide a clear answer as to how different methods are to be compared. Focusing on text classification and the model debugging scenario, our main contribution is a protocol for faithfulness evaluation that makes use of partially synthetic data to obtain ground truth for feature importance ranking. Following the protocol, we do an in-depth analysis of four standard salience method classes on a range of datasets and shortcuts for BERT and LSTM models and demonstrate that some of the most popular method configurations provide poor results even for simplest shortcuts. We recommend following the protocol for each new task and model combination to find the best method for identifying shortcuts.

Key facts. Cited by 66 papers in the map (data through 7 Oct 2026). Builds on 46 of its 67 references that are in the map. First posted on arXiv 14 Nov 2021.

Builds on

46 of 67 references.

References · 46

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  3. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  4. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  5. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  6. 2019Attention is not Explanationcited by 916
  7. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  8. 2019Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)cited by 657
  9. 2018Annotation Artifacts in Natural Language Inference Datacited by 656
  10. 2020Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?cited by 590
  11. 2019ERASER: A Benchmark to Evaluate Rationalized NLP Modelscited by 523
  12. 2018A Benchmark for Interpretability Methods in Deep Neural Networkscited by 467
34 more
  1. 2019Learning the Difference that Makes a Difference with Counterfactually-Augmented Datacited by 428
  2. 2020Generating Radiology Reports via Memory-driven Transformercited by 363
  3. 2017The (Un)reliability of saliency methodscited by 331
  4. 2019On the (In)fidelity and Sensitivity for Explanationscited by 309
  5. 2018Hypothesis Only Baselines in Natural Language Inferencecited by 291
  6. 2016Ex Machina: Personal Attacks Seen at Scalecited by 208
  7. 2015Visualizing and Understanding Neural Models in NLPcited by 199
  8. 2019Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasetscited by 173
  9. 2020A Diagnostic Study of Explainability Techniques for Text Classificationcited by 158
  10. 2020Debugging Tests for Model Explanationscited by 146
  11. 2020Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functionscited by 144
  12. 2020The Language Interpretability Tool: Extensible, Interactive Visualizations and Analysis for NLP Modelscited by 125
  13. 2020The elephant in the interpretability room: Why use attention as explanation when we have saliency methods?cited by 122
  14. 2020Explaining NLP Models via Minimal Contrastive Editing (MiCE)cited by 105
  15. 2021Do Feature Attribution Methods Correctly Attribute Features?cited by 98
  16. 2019Benchmarking Attribution Methods with Relative Feature Importancecited by 89
  17. 2021Towards Interpreting and Mitigating Shortcut Learning Behavior of NLU Modelscited by 78
  18. 2018Did the Model Understand the Question?cited by 60
  19. 2018What can AI do for me: Evaluating Machine Learning Interpretations in Cooperative Playcited by 52
  20. 2014Extraction of Salient Sentences from Labelled Documentscited by 50
  21. 2019Quantifying Interpretability and Trust in Machine Learning Systemscited by 49
  22. 2019Don't Take the Premise for Granted: Mitigating Artifacts in Natural Language Inferencecited by 48
  23. 2020Interpretation of NLP models through input marginalizationcited by 43
  24. 2021Combining Feature and Instance Attribution to Detect Artifactscited by 38
  25. 2019Can I Trust the Explainer? Verifying Post-hoc Explanatory Methodscited by 37
  26. 2021Evaluating Saliency Methods for Neural Language Modelscited by 37
  27. 2019Do Human Rationales Improve Machine Explanations?cited by 33
  28. 2021Evaluating the Faithfulness of Importance Measures in NLP by Recursively Masking Allegedly Important Tokens and Retrainingcited by 31
  29. 2021On the Sensitivity and Stability of Model Interpretations in NLPcited by 30
  30. 2019Evaluating Recurrent Neural Network Explanationscited by 29
  31. 2020Exposing Shallow Heuristics of Relation Extraction Models with Challenge Datacited by 20
  32. 2020The Struggles of Feature-Based Explanations: Shapley Values vs. Minimal Sufficient Subsetscited by 14
  33. 2021Towards Benchmarking the Utility of Explanations for Model Debuggingcited by 10
  34. 2020Evaluating Attribution Methods using White-Box LSTMscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 15; 2023: 23; 2024: 15; 2025: 10; 2026: 3.

66 papers.

Sort:

By area:

  1. 2023Explainability for Large Language Models: A Surveycited by 331
  2. 2023Protein Design with Guided Discrete Diffusioncited by 154
  3. 2022Towards Faithful Model Explanation in NLP: A Surveycited by 129
  4. 2022Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networkscited by 126
  5. 2024Rethinking Interpretability in the Era of Large Language Modelscited by 110
  6. 2024Are self-explanations from Large Language Models faithful?cited by 101
  7. 2023Can Large Language Models Explain Themselves? A Study of LLM-Generated Self-Explanationscited by 90
  8. 2023Inseq: An Interpretability Toolkit for Sequence Generation Modelscited by 78
  9. 2022Measuring the Mixing of Contextual Information in the Transformercited by 63
  10. 2022Post hoc Explanations may be Ineffective for Detecting Unknown Spurious Correlationcited by 59
  11. 2024NNsight and NDIF: Democratizing Access to Open-Weight Foundation Model Internalscited by 43
  12. 2023DecompX: Explaining Transformers Decisions by Propagating Token Decompositioncited by 31
38 more
  1. 2024Model Internals-based Answer Attribution for Trustworthy Retrieval-Augmented Generationcited by 25
  2. 2022Learning to Scaffold: Optimizing Model Explanations for Teachingcited by 20
  3. 2022Diagnosing AI Explanation Methods with Folk Concepts of Behaviorcited by 15
  4. 2022Finding Dataset Shortcuts with Grammar Inductioncited by 15
  5. 2023The Inside Story: Towards Better Understanding of Machine Translation Neural Evaluation Metricscited by 15
  6. 2023Are Character-level Translations Worth the Wait? Comparing ByT5 and mT5 for Machine Translationcited by 14
  7. 2023Stability Guarantees for Feature Attributions with Multiplicative Smoothingcited by 14
  8. 2023Fuse to Forget: Bias Reduction and Selective Memorization through Model Fusioncited by 13
  9. 2022The Solvability of Interpretability Evaluation Metricscited by 11
  10. 2024Computational Experiments Meet Large Language Model Based Agents: A Survey and Perspectivecited by 11
  11. 2023Transparency at the Source: Evaluating and Interpreting Language Models With Access to the True Distributioncited by 10
  12. 2024Interpretability Needs a New Paradigmcited by 10
  13. 2025A Close Look at Decomposition-based XAI-Methods for Transformer Language Modelscited by 10
  14. 2023On the Impact of Knowledge Distillation for Model Interpretabilitycited by 8
  15. 2023Feature Interactions Reveal Linguistic Structure in Language Modelscited by 8
  16. 2023Adversarial Attacks on the Interpretation of Neuron Activation Maximizationcited by 7
  17. 2025Randomly Removing 50% of Dimensions in Text Embeddings has Minimal Impact on Retrieval and Classification Taskscited by 7
  18. 2022Assessing Out-of-Domain Language Model Performance from Few Examplescited by 6
  19. 2023Quantifying the Plausibility of Context Reliance in Neural Machine Translationcited by 6
  20. 2024Comparing Explanation Faithfulness between Multilingual and Monolingual Fine-tuned Language Modelscited by 6
  21. 2024Interactive Prompt Debugging with Sequence Saliencecited by 6
  22. 2022Auditing Visualizations: Transparency Methods Struggle to Detect Anomalous Behaviorcited by 5
  23. 2023Faithfulness Measurable Masked Language Modelscited by 5
  24. 2023Contrastive Learning-based Sentence Encoders Implicitly Weight Informative Wordscited by 5
  25. 2025Short-circuiting Shortcuts: Mechanistic Investigation of Shortcuts in Text Classificationcited by 5
  26. 2022When less is more: Simplifying inputs aids neural network understandingcited by 4
  27. 2022Cascading Biases: Investigating the Effect of Heuristic Annotation Strategies on Data and Modelscited by 4
  28. 2023A Multi-Grained Self-Interpretable Symbolic-Neural Model For Single/Multi-Labeled Text Classificationcited by 4
  29. 2023Dynamic Top-k Estimation Consolidates Disagreement between Feature Attribution Methodscited by 4
  30. 2023Attribution and Alignment: Effects of Local Context Repetition on Utterance Production and Comprehension in Dialoguecited by 4
  31. 2023Accelerating the Global Aggregation of Local Explanationscited by 4
  32. 2023Towards Faithful Explanations for Text Classification with Robustness Improvement and Explanation Guided Trainingcited by 4
  33. 2025Explanations of Large Language Models Explain Language Representations in the Braincited by 4
  34. 2025Explainable Compliance Detection with Multi-Hop Natural Language Inference on Assurance Case Structurecited by 4
  35. 2022Assessing the Impact of Sequence Length Learning on Classification Tasks for Transformer Encoder Modelscited by 3
  36. 2024The Role of Syntactic Span Preferences in Post-Hoc Explanation Disagreementcited by 3
  37. 2024Counterfactuals As a Means for Evaluating Faithfulness of Attribution Methods in Autoregressive Language Modelscited by 3
  38. 2024Explanations that reveal all through the definition of encodingcited by 3

50 most cited of 66 All 66 (JSON)