# A Song of (Dis)agreement: Evaluating the Evaluation of Explainable Artificial Intelligence in Natural Language Processing arXiv: https://arxiv.org/abs/2205.04559 ยท HTML: https://mlsanity.com/paper/2205.04559/ ## Key facts - Cited by 13 papers in the map (data through 14 Oct 2026). - Builds on 46 of its 59 references that are in the map. - First posted on arXiv 9 May 2022. ## Builds on (most cited first, 30 of 46) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014) - [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017) - ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016) - [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014) - [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019) - [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017) - [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015) - [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016) - [Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1704.02685/) (2017) - [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015) - [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017) - [What Does BERT Look At? An Analysis of BERT's Attention](https://mlsanity.com/paper/1906.04341/) (2019) - [Attention is not Explanation](https://mlsanity.com/paper/1902.10186/) (2019) - [XNLI: Evaluating Cross-lingual Sentence Representations](https://mlsanity.com/paper/1809.05053/) (2018) - [Quantifying Attention Flow in Transformers](https://mlsanity.com/paper/2005.00928/) (2020) - [Attention is not not Explanation](https://mlsanity.com/paper/1908.04626/) (2019) - [Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?](https://mlsanity.com/paper/2004.03685/) (2020) - [ERASER: A Benchmark to Evaluate Rationalized NLP Models](https://mlsanity.com/paper/1911.03429/) (2019) - [Is Attention Interpretable?](https://mlsanity.com/paper/1906.03731/) (2019) - [AllenNLP: A Deep Semantic Natural Language Processing Platform](https://mlsanity.com/paper/1803.07640/) (2018) - [Understanding Neural Networks through Representation Erasure](https://mlsanity.com/paper/1612.08220/) (2016) - [Visualizing and Understanding Neural Models in NLP](https://mlsanity.com/paper/1506.01066/) (2015) - [A Diagnostic Study of Explainability Techniques for Text Classification](https://mlsanity.com/paper/2009.13295/) (2020) - [The Disagreement Problem in Explainable Machine Learning: A Practitioner's Perspective](https://mlsanity.com/paper/2202.01602/) (2022) - [The elephant in the interpretability room: Why use attention as explanation when we have saliency methods?](https://mlsanity.com/paper/2010.05607/) (2020) - [On Identifiability in Transformers](https://mlsanity.com/paper/1908.04211/) (2019) - [TimeSHAP: Explaining Recurrent Models through Sequence Perturbations](https://mlsanity.com/paper/2012.00073/) (2020) - [Attention Interpretability Across NLP Tasks](https://mlsanity.com/paper/1909.11218/) (2019) ## Built on by (most cited first, 13 of 13) - [Quantifying Context Mixing in Transformers](https://mlsanity.com/paper/2301.12971/) (2023) - [DecoderLens: Layerwise Interpretation of Encoder-Decoder Transformers](https://mlsanity.com/paper/2310.03686/) (2023) - [CATfOOD: Counterfactual Augmented Training for Improving Out-of-Domain Performance and Calibration](https://mlsanity.com/paper/2309.07822/) (2023) - [Attention Meets Post-hoc Interpretability: A Mathematical Perspective](https://mlsanity.com/paper/2402.03485/) (2024) - [Feature Interactions Reveal Linguistic Structure in Language Models](https://mlsanity.com/paper/2306.12181/) (2023) - [Dynamic Top-k Estimation Consolidates Disagreement between Feature Attribution Methods](https://mlsanity.com/paper/2310.05619/) (2023) - [Fake News Detection After LLM Laundering: Measurement and Explanation](https://mlsanity.com/paper/2501.18649/) (2025) - [The Role of Syntactic Span Preferences in Post-Hoc Explanation Disagreement](https://mlsanity.com/paper/2403.19424/) (2024) - [Semantic match: Debugging feature attribution methods in XAI for healthcare](https://mlsanity.com/paper/2301.02080/) (2023) - [Fixing confirmation bias in feature attribution methods via semantic match](https://mlsanity.com/paper/2307.00897/) (2023) - [On the reliability of feature attribution methods for speech classification](https://mlsanity.com/paper/2505.16406/) (2025) - [Black Big Boxes: Tracing Adjective Order Preferences in Large Language Models](https://mlsanity.com/paper/2407.02136/) (2024) - ["Let's Agree to Disagree": Investigating the Disagreement Problem in Explainable AI for Text Summarization](https://mlsanity.com/paper/2410.18560/) (2024) ## Abstract (arXiv) There has been significant debate in the NLP community about whether or not attention weights can be used as an explanation - a mechanism for interpreting how important each input token is for a particular prediction. The validity of "attention as explanation" has so far been evaluated by computing the rank correlation between attention-based explanations and existing feature attribution explanations using LSTM-based models. In our work, we (i) compare the rank correlation between five more recent feature attribution methods and two attention-based methods, on two types of NLP tasks, and (ii) extend this analysis to also include transformer-based models. We find that attention-based explanations do not correlate strongly with any recent feature attribution methods, regardless of the model or task. Furthermore, we find that none of the tested explanations correlate strongly with one another for the transformer-based model, leading us to question the underlying assumption that we should measure the validity of attention-based explanations based on how well they correlate with existing feature attribution explanation methods. After conducting experiments on five datasets using two different models, we argue that the community should stop using rank correlation as an evaluation metric for attention-based explanations. We suggest that researchers and practitioners should instead test various explanation methods and employ a human-in-the-loop process to determine if the explanations align with human intuition for the particular use case at hand.