# SEAT: Stable and Explainable Attention

arXiv: https://arxiv.org/abs/2211.13290 · HTML: https://mlsanity.com/paper/2211.13290/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 42 references that are in the map.
- First posted on arXiv 23 Nov 2022.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Attention is not Explanation](https://mlsanity.com/paper/1902.10186/) (2019)
- [Attention is not not Explanation](https://mlsanity.com/paper/1908.04626/) (2019)
- [Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?](https://mlsanity.com/paper/2004.03685/) (2020)
- [ERASER: A Benchmark to Evaluate Rationalized NLP Models](https://mlsanity.com/paper/1911.03429/) (2019)
- [Interpretation of Neural Networks is Fragile](https://mlsanity.com/paper/1710.10547/) (2017)
- [Is Attention Interpretable?](https://mlsanity.com/paper/1906.03731/) (2019)
- [RETAIN: An Interpretable Predictive Model for Healthcare using Reverse Time Attention Mechanism](https://mlsanity.com/paper/1608.05745/) (2016)
- [From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification](https://mlsanity.com/paper/1602.02068/) (2016)
- [On the (In)fidelity and Sensitivity for Explanations](https://mlsanity.com/paper/1901.09392/) (2019)
- [Adversarial Training Methods for Semi-Supervised Text Classification](https://mlsanity.com/paper/1605.07725/) (2016)
- [Techniques for Interpretable Machine Learning](https://mlsanity.com/paper/1808.00033/) (2018)
- [Explanations can be manipulated and geometry is to blame](https://mlsanity.com/paper/1906.07983/) (2019)
- [Attention in Natural Language Processing](https://mlsanity.com/paper/1902.02181/) (2019)
- [Interpretable Adversarial Perturbation in Input Embedding Space for Text](https://mlsanity.com/paper/1805.02917/) (2018)
- [EditNTS: An Neural Programmer-Interpreter Model for Sentence Simplification through Explicit Editing](https://mlsanity.com/paper/1906.08104/) (2019)
- [Towards Transparent and Explainable Attention Models](https://mlsanity.com/paper/2004.14243/) (2020)
- [On the Sensitivity and Stability of Model Interpretations in NLP](https://mlsanity.com/paper/2104.08782/) (2021)
- [Attention Meets Perturbations: Robust and Interpretable Attention with Adversarial Training](https://mlsanity.com/paper/2009.12064/) (2020)

## Built on by (most cited first, 25 of 25)

- [A Survey of Mamba](https://mlsanity.com/paper/2408.01129/) (2024)
- [Multi-hop Question Answering under Temporal Knowledge Editing](https://mlsanity.com/paper/2404.00492/) (2024)
- [Editable Concept Bottleneck Models](https://mlsanity.com/paper/2405.15476/) (2024)
- [Semi-supervised Concept Bottleneck Models](https://mlsanity.com/paper/2406.18992/) (2024)
- [Prompt-SAW: Leveraging Relation-Aware Graphs for Textual Prompt Compression](https://mlsanity.com/paper/2404.00489/) (2024)
- [A Comprehensive Survey on Self-Interpretable Neural Networks](https://mlsanity.com/paper/2501.15638/) (2025)
- [Improving Interpretation Faithfulness for Vision Transformers](https://mlsanity.com/paper/2311.17983/) (2023)
- [Leveraging Logical Rules in Knowledge Editing: A Cherry on the Top](https://mlsanity.com/paper/2405.15452/) (2024)
- [Dialectical Alignment: Resolving the Tension of 3H and Security Threats of LLMs](https://mlsanity.com/paper/2404.00486/) (2024)
- [Understanding Reasoning in Chain-of-Thought from the Hopfieldian View](https://mlsanity.com/paper/2410.03595/) (2024)
- [Adaptive Multi-prompt Contrastive Network for Few-shot Out-of-distribution Detection](https://mlsanity.com/paper/2506.17633/) (2025)
- [MONAL: Model Autophagy Analysis for Modeling Human-AI Interactions](https://mlsanity.com/paper/2402.11271/) (2024)
- [FTS: A Framework to Find a Faithful TimeSieve](https://mlsanity.com/paper/2405.19647/) (2024)
- [Source Attribution in Retrieval-Augmented Generation](https://mlsanity.com/paper/2507.04480/) (2025)
- [Explainable AI for Mental Disorder Detection via Social Media: A survey and outlook](https://mlsanity.com/paper/2406.05984/) (2024)
- [Towards Multi-dimensional Explanation Alignment for Medical Classification](https://mlsanity.com/paper/2410.21494/) (2024)
- [DRIVE: Dependable Robust Interpretable Visionary Ensemble Framework in Autonomous Driving](https://mlsanity.com/paper/2409.10330/) (2024)
- [Benchmarking and Mitigating Sycophancy in Medical Vision Language Models](https://mlsanity.com/paper/2509.21979/) (2025)
- [Faithful Interpretation for Graph Neural Networks](https://mlsanity.com/paper/2410.06950/) (2024)
- [Stable Vision Concept Transformers for Medical Diagnosis](https://mlsanity.com/paper/2506.05286/) (2025)
- [The Compositional Architecture of Regret in Large Language Models](https://mlsanity.com/paper/2506.15617/) (2025)
- [Controllable Concept Bottleneck Models](https://mlsanity.com/paper/2601.00451/) (2026)
- [Efficient Text-Attributed Graph Learning through Selective Annotation and Graph Alignment](https://mlsanity.com/paper/2506.07168/) (2025)
- [Empirical Characterization of Rationale Stability Under Controlled Perturbations for Explainable Pattern Recognition](https://mlsanity.com/paper/2604.04456/) (2026)
- [Hyperbolic Concept Bottleneck Models](https://mlsanity.com/paper/2605.06440/) (2026)

## Abstract (arXiv)

Currently, attention mechanism becomes a standard fixture in most state-of-the-art natural language processing (NLP) models, not only due to outstanding performance it could gain, but also due to plausible innate explanation for the behaviors of neural architectures it provides, which is notoriously difficult to analyze. However, recent studies show that attention is unstable against randomness and perturbations during training or testing, such as random seeds and slight perturbation of embedding vectors, which impedes it from becoming a faithful explanation tool. Thus, a natural question is whether we can find some substitute of the current attention which is more stable and could keep the most important characteristics on explanation and prediction of attention. In this paper, to resolve the problem, we provide a first rigorous definition of such alternate namely SEAT (Stable and Explainable Attention). Specifically, a SEAT should has the following three properties: (1) Its prediction distribution is enforced to be close to the distribution based on the vanilla attention; (2) Its top-k indices have large overlaps with those of the vanilla attention; (3) It is robust w.r.t perturbations, i.e., any slight perturbation on SEAT will not change the prediction distribution too much, which implicitly indicates that it is stable to randomness and perturbations. Finally, through intensive experiments on various datasets, we compare our SEAT with other baseline methods using RNN, BiLSTM and BERT architectures via six different evaluation metrics for model interpretation, stability and accuracy. Results show that SEAT is more stable against different perturbations and randomness while also keeps the explainability of attention, which indicates it is a more faithful explanation. Moreover, compared with vanilla attention, there is almost no utility (accuracy) degradation for SEAT.