mlsanityData to 14 Oct 2026

Order in the Court: Explainable AI Methods Prone to Disagreement

arXiv 2105.03287 ↗First posted 7 May 2021cs.LG cs.CLcited by 36

Not on the map. Why?

Abstract (arXiv)

By computing the rank correlation between attention weights and feature-additive explanation methods, previous analyses either invalidate or support the role of attention-based explanations as a faithful and plausible measure of salience. To investigate whether this approach is appropriate, we compare LIME, Integrated Gradients, DeepLIFT, Grad-SHAP, Deep-SHAP, and attention-based explanations, applied to two neural architectures trained on single- and pair-sequence language tasks. In most cases, we find that none of our chosen methods agree. Based on our empirical observations and theoretical objections, we conclude that rank correlation does not measure the quality of feature-additive methods. Practitioners should instead use the numerous and rigorous diagnostic methods proposed by the community.

Key facts. Cited by 36 papers in the map (data through 14 Oct 2026). Builds on 43 of its 64 references that are in the map. First posted on arXiv 7 May 2021.

Builds on

43 of 64 references.

References · 43

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
  5. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,070
  6. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  8. 2017Axiomatic Attribution for Deep Networkscited by 3,384
  9. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  10. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  11. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  12. 2016Enriching Word Vectors with Subword Informationcited by 1,583
31 more
  1. 2017Learning Important Features Through Propagating Activation Differencescited by 1,458
  2. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,416
  3. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
  4. 2019Attention is not Explanationcited by 929
  5. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  6. 2020Quantifying Attention Flow in Transformerscited by 775
  7. 2019Attention is not not Explanationcited by 636
  8. 2020Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?cited by 599
  9. 2019ERASER: A Benchmark to Evaluate Rationalized NLP Modelscited by 530
  10. 2018A Benchmark for Interpretability Methods in Deep Neural Networkscited by 476
  11. 2018e-SNLI: Natural Language Inference with Natural Language Explanationscited by 468
  12. 2018AllenNLP: A Deep Semantic Natural Language Processing Platformcited by 347
  13. 2017The (Un)reliability of saliency methodscited by 333
  14. 2016Understanding Neural Networks through Representation Erasurecited by 224
  15. 2020A Diagnostic Study of Explainability Techniques for Text Classificationcited by 159
  16. 2019Adaptively Sparse Transformerscited by 155
  17. 2020The elephant in the interpretability room: Why use attention as explanation when we have saliency methods?cited by 122
  18. 2019On Identifiability in Transformerscited by 116
  19. 2019Learning to Deceive with Attention-Based Explanationscited by 111
  20. 2020Explaining Explanations: Axiomatic Feature Interactions for Deep Networkscited by 103
  21. 2019Attention in Natural Language Processingcited by 101
  22. 2020TimeSHAP: Explaining Recurrent Models through Sequence Perturbationscited by 74
  23. 2016Investigating the influence of noise and distractors on the interpretation of neural networkscited by 54
  24. 2020Towards Transparent and Explainable Attention Modelscited by 47
  25. 2018Interpretable Credit Application Predictions With Counterfactual Explanationscited by 43
  26. 2019Can I Trust the Explainer? Verifying Post-hoc Explanatory Methodscited by 38
  27. 2016Interpretation of Prediction Models Using the Input Gradientcited by 24
  28. 2019Fine-grained Sentiment Analysis with Faithful Attentioncited by 23
  29. 2019On the Convergence Proof of AMSGrad and a New Versioncited by 21
  30. 2020Staying True to Your Word: (How) Can Attention Become Explanation?cited by 18
  31. 2019Understanding Multi-Head Attention in Abstractive Summarizationcited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 4; 2022: 7; 2023: 7; 2024: 10; 2025: 4; 2026: 4.

36 papers.

Sort:

By area:

  1. 2023Explainability for Large Language Models: A Surveycited by 333
  2. 2022The Disagreement Problem in Explainable Machine Learning: A Practitioner's Perspectivecited by 159
  3. 2023The Meta-Evaluation Problem in Explainable AI: Identifying Reliable Estimators with MetaQuantuscited by 37
  4. 2024Decoupling Pixel Flipping and Occlusion Strategy for Consistent XAI Benchmarkscited by 28
  5. 2023An Empirical Evaluation of the Rashomon Effect in Explainable Machine Learningcited by 25
  6. 2022Learning to Scaffold: Optimizing Model Explanations for Teachingcited by 20
  7. 2022A Song of (Dis)agreement: Evaluating the Evaluation of Explainable Artificial Intelligence in Natural Language Processingcited by 13
  8. 2021Thermostat: A Large Collection of NLP Model Explanations and Analysis Toolscited by 11
  9. 2022Explainability in reinforcement learning: perspective and positioncited by 11
  10. 2024Attention Meets Post-hoc Interpretability: A Mathematical Perspectivecited by 11
  11. 2023Manipulation Risks in Explainable AI: The Implications of the Disagreement Problemcited by 7
  12. 2024Toward Understanding the Disagreement Problem in Neural Network Feature Attributioncited by 6
24 more
  1. 2024Why do explanations fail? A typology and discussion on failures in XAIcited by 6
  2. 2024Provably Better Explanations with Optimized Aggregation of Feature Attributionscited by 6
  3. 2021Seven challenges for harmonizing explainability requirementscited by 4
  4. 2021Reproducibility as a Mechanism for Teaching Fairness, Accountability, Confidentiality, and Transparency in Artificial Intelligencecited by 4
  5. 2025Aligning What LLMs Do and Say: Towards Self-Consistent Explanationscited by 4
  6. 2022Reducing Predictive Feature Suppression in Resource-Constrained Contrastive Image-Caption Retrievalcited by 3
  7. 2023Disagreement amongst counterfactual explanations: How transparency can be deceptivecited by 3
  8. 2024Prototype-Based Methods in Explainable AI and Emerging Opportunities in the Geosciencescited by 3
  9. 2021Combining Transformers with Natural Language Explanationscited by 1
  10. 2023Is Task-Agnostic Explainable AI a Myth?cited by 1
  11. 2024Backward Compatibility in Attributive Explanation and Enhanced Model Training Methodcited by 1
  12. 2024Explainability in AI Based Applications: A Framework for Comparing Different Techniquescited by 1
  13. 2025Fast Fourier Correlation is a Highly Efficient and Accurate Feature Attribution Algorithm from the Perspective of Control Theory and Game Theorycited by 1
  14. 2022Accountable and Explainable Methods for Complex Reasoning over Textcited by 0
  15. 2022Easy to Decide, Hard to Agree: Reducing Disagreements Between Saliency Methodscited by 0
  16. 2023Five policy uses of algorithmic transparency and explainabilitycited by 0
  17. 2024Extracting Explanations, Justification, and Uncertainty from Black-Box Deep Neural Networkscited by 0
  18. 2024Comprehensive and Reliable Feature Attribution for Diverse Modalities and Models via Frequency-Domain Insightscited by 0
  19. 2025Feature Importance Depends on Properties of the Data: Towards Choosing the Correct Explanations for Your Data and Decision Trees based Modelscited by 0
  20. 2025How can we trust opaque systems? Criteria for robust explanations in XAIcited by 0
  21. 2026On the Definition and Detection of Cherry-Picking in Counterfactual Explanationscited by 0
  22. 2026Prototype-based Explainable Neural Networks with Channel-specific Reasoning for Geospatial Learning Taskscited by 0
  23. 2026The Impact of Stochasticity on the Rashomon Effect in Machine Learningcited by 0
  24. 2026Mask-Induced Displacement in Audio XAI via Logit Trajectory Decompositioncited by 0