# Do Transformer Attention Heads Provide Transparency in Abstractive Summarization?

arXiv: https://arxiv.org/abs/1907.00570 · HTML: https://mlsanity.com/paper/1907.00570/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Jul 2019.

## Built on by (most cited first, 6 of 6)

- [Zorro: Valid, Sparse, and Stable Explanations in Graph Neural Networks](https://mlsanity.com/paper/2105.08621/) (2021)
- [Attention Approximates Sparse Distributed Memory](https://mlsanity.com/paper/2111.05498/) (2021)
- [Diagnosing AI Explanation Methods with Folk Concepts of Behavior](https://mlsanity.com/paper/2201.11239/) (2022)
- [Towards explainable evaluation of language models on the semantic similarity of visual concepts](https://mlsanity.com/paper/2209.03723/) (2022)
- [GADformer: A Transparent Transformer Model for Group Anomaly Detection on Trajectories](https://mlsanity.com/paper/2303.09841/) (2023)
- [Extracting Interpretable Local and Global Representations from Attention on Time Series](https://mlsanity.com/paper/2312.11466/) (2023)

## Abstract (arXiv)

Learning algorithms become more powerful, often at the cost of increased complexity. In response, the demand for algorithms to be transparent is growing. In NLP tasks, attention distributions learned by attention-based deep learning models are used to gain insights in the models' behavior. To which extent is this perspective valid for all NLP tasks? We investigate whether distributions calculated by different attention heads in a transformer architecture can be used to improve transparency in the task of abstractive summarization. To this end, we present both a qualitative and quantitative analysis to investigate the behavior of the attention heads. We show that some attention heads indeed specialize towards syntactically and semantically distinct input. We propose an approach to evaluate to which extent the Transformer model relies on specifically learned attention distributions. We also discuss what this implies for using attention distributions as a means of transparency.