mlsanityData to 14 Oct 2026

The elephant in the interpretability room: Why use attention as explanation when we have saliency methods?

arXiv 2010.05607 ↗First posted 12 Oct 2020cs.CLcited by 122

Not on the map. Why?

Abstract (arXiv)

There is a recent surge of interest in using attention as explanation of model predictions, with mixed evidence on whether attention can be used as such. While attention conveniently gives us one weight per input token and is easily extracted, it is often unclear toward what goal it is used as explanation. We find that often that goal, whether explicitly stated or not, is to find out what input tokens are the most relevant to a prediction, and that the implied user for the explanation is a model developer. For this goal and user, we argue that input saliency methods are better suited, and that there are no compelling reasons to use attention, despite the coincidence that it provides a weight for each input. With this position paper, we hope to shift some of the recent focus on attention to saliency methods, and for authors to clearly state the goal and user for their explanations.

Key facts. Cited by 122 papers in the map (data through 14 Oct 2026). First posted on arXiv 12 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 28; 2022: 25; 2023: 23; 2024: 16; 2025: 12; 2026: 18.

122 papers.

Sort:

By area:

  1. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 528
  2. 2021Post-hoc Interpretability for Neural NLP: A Surveycited by 156
  3. 2022Towards Faithful Model Explanation in NLP: A Surveycited by 130
  4. 2023AI Transparency in the Age of LLMs: A Human-Centered Research Roadmapcited by 123
  5. 2024Information Flow Routes: Automatically Interpreting Language Models at Scalecited by 95
  6. 2023Attention Sorting Combats Recency Bias In Long Context Language Modelscited by 90
  7. 2022Multi-Modal Knowledge Graph Construction and Application: A Surveycited by 74
  8. 2021"Will You Find These Shortcuts?" A Protocol for Evaluating the Faithfulness of Input Salience Methods for Text Classificationcited by 66
  9. 2022Looking for a Needle in a Haystack: A Comprehensive Study of Hallucinations in Neural Machine Translationcited by 61
  10. 2022Leveraging Explanations in Interactive Machine Learning: An Overviewcited by 53
  11. 2022polyBERT: A chemical language model to enable fully machine-driven ultrafast polymer informaticscited by 48
  12. 2022GlobEnc: Quantifying Global Token Attribution by Incorporating the Whole Encoder Layer in Transformerscited by 44
38 more
  1. 2022Grad-SAM: Explaining Transformers via Gradient Self-Attention Mapscited by 42
  2. 2021Evaluating Saliency Methods for Neural Language Modelscited by 37
  3. 2021Order in the Court: Explainable AI Methods Prone to Disagreementcited by 36
  4. 2021Understanding Interlocking Dynamics of Cooperative Rationalizationcited by 36
  5. 2023HalOmi: A Manually Annotated Benchmark for Multilingual Hallucination and Omission Detection in Machine Translationcited by 35
  6. 2022SBERT studies Meaning Representations: Decomposing Sentence Embeddings into Explainable Semantic Featurescited by 33
  7. 2021UNIREX: A Unified Learning Framework for Language Model Rationale Extractioncited by 32
  8. 2021Is Sparse Attention more Interpretable?cited by 31
  9. 2021Evaluating the Faithfulness of Importance Measures in NLP by Recursively Masking Allegedly Important Tokens and Retrainingcited by 31
  10. 2023ConvXAI: Delivering Heterogeneous AI Explanations via Conversations to Support Human-AI Scientific Writingcited by 31
  11. 2023DecompX: Explaining Transformers Decisions by Propagating Token Decompositioncited by 31
  12. 2023B-cos Alignment for Inherently Interpretable CNNs and Vision Transformerscited by 31
  13. 2021On the Sensitivity and Stability of Model Interpretations in NLPcited by 30
  14. 2021Exploring the Role of BERT Token Representations to Explain Sentence Probing Resultscited by 29
  15. 2021Multilingual Language Models Predict Human Reading Behaviorcited by 29
  16. 2022ViT-CX: Causal Explanation of Vision Transformerscited by 28
  17. 2023Using Sequences of Life-events to Predict Human Livescited by 27
  18. 2021Rationales for Sequential Predictionscited by 26
  19. 2023AD-KD: Attribution-Driven Knowledge Distillation for Language Model Compressioncited by 26
  20. 2023Opti-CAM: Optimizing saliency maps for interpretabilitycited by 25
  21. 2022Deconfounding Legal Judgment Prediction for European Court of Human Rights Cases Towards Better Alignment with Expertscited by 24
  22. 2021Relative Importance in Sentence Processingcited by 22
  23. 2022AdapLeR: Speeding up Inference by Adaptive Length Reductioncited by 19
  24. 2024On the Faithfulness of Vision Transformer Explanationscited by 19
  25. 2021Enjoy the Salience: Towards Better Transformer-based Faithful Explanations with Word Saliencecited by 16
  26. 2024LM Transparency Tool: Interactive Tool for Analyzing Transformer Language Modelscited by 16
  27. 2021Counterfactual Evaluation for Explainable AIcited by 15
  28. 2023Incorporating Attribution Importance for Improving Faithfulness Metricscited by 14
  29. 2022A Song of (Dis)agreement: Evaluating the Evaluation of Explainable Artificial Intelligence in Natural Language Processingcited by 13
  30. 2024ReAGent: A Model-agnostic Feature Attribution Method for Generative Language Modelscited by 12
  31. 2024Attention Meets Post-hoc Interpretability: A Mathematical Perspectivecited by 11
  32. 2024Interpretability Needs a New Paradigmcited by 11
  33. 2025The Atlas of In-Context Learning: How Attention Heads Shape In-Context Retrieval Augmentationcited by 11
  34. 2021Flexible Instance-Specific Rationalization of NLP Modelscited by 10
  35. 2021Dissecting Generation Modes for Abstractive Summarization Models via Ablation and Attributioncited by 10
  36. 2021CVSS-BERT: Explainable Natural Language Processing to Determine the Severity of a Computer Security Vulnerability from its Descriptioncited by 10
  37. 2024Normalized AOPC: Fixing Misleading Faithfulness Metrics for Feature Attribution Explainabilitycited by 10
  38. 2024Detecting Hallucination and Coverage Errors in Retrieval Augmented Generation for Controversial Topicscited by 9

50 most cited of 122 All 122 (JSON)