# How Much Does Attention Actually Attend? Questioning the Importance of Attention in Pretrained Transformers

arXiv: https://arxiv.org/abs/2211.03495 · HTML: https://mlsanity.com/paper/2211.03495/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 43 references that are in the map.
- First posted on arXiv 7 Nov 2022.

## Builds on (most cited first, 30 of 35)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [What Does BERT Look At? An Analysis of BERT's Attention](https://mlsanity.com/paper/1906.04341/) (2019)
- [Neural Network Acceptability Judgments](https://mlsanity.com/paper/1805.12471/) (2018)
- [Attention is not Explanation](https://mlsanity.com/paper/1902.10186/) (2019)
- [Probing Classifiers: Promises, Shortcomings, and Advances](https://mlsanity.com/paper/2102.12452/) (2021)
- [Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned](https://mlsanity.com/paper/1905.09418/) (2019)
- [Are Sixteen Heads Really Better than One?](https://mlsanity.com/paper/1905.10650/) (2019)
- [Attention is not not Explanation](https://mlsanity.com/paper/1908.04626/) (2019)
- [Is Attention Interpretable?](https://mlsanity.com/paper/1906.03731/) (2019)
- [Linguistic Knowledge and Transferability of Contextual Representations](https://mlsanity.com/paper/1903.08855/) (2019)
- [Pay Attention to MLPs](https://mlsanity.com/paper/2105.08050/) (2021)
- [Random Feature Attention](https://mlsanity.com/paper/2103.02143/) (2021)
- [FNet: Mixing Tokens with Fourier Transforms](https://mlsanity.com/paper/2105.03824/) (2021)
- [cosFormer: Rethinking Softmax in Attention](https://mlsanity.com/paper/2202.08791/) (2022)
- [Transformer Quality in Linear Time](https://mlsanity.com/paper/2202.10447/) (2022)
- [Finetuning Pretrained Transformers into RNNs](https://mlsanity.com/paper/2103.13076/) (2021)
- [Do Attention Heads in BERT Track Syntactic Dependencies?](https://mlsanity.com/paper/1911.12246/) (2019)
- [Differentiable Subset Pruning of Transformer Heads](https://mlsanity.com/paper/2108.04657/) (2021)

## Built on by (most cited first, 21 of 21)

- [PMET: Precise Model Editing in a Transformer](https://mlsanity.com/paper/2308.08742/) (2023)
- [The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networks](https://mlsanity.com/paper/2306.17844/) (2023)
- [Transformers are Multi-State RNNs](https://mlsanity.com/paper/2401.06104/) (2024)
- [Attention Score is not All You Need for Token Importance Indicator in KV Cache Reduction: Value Also Matters](https://mlsanity.com/paper/2406.12335/) (2024)
- [Transformers Learn to Achieve Second-Order Convergence Rates for In-Context Linear Regression](https://mlsanity.com/paper/2310.17086/) (2023)
- [Quantifying Context Mixing in Transformers](https://mlsanity.com/paper/2301.12971/) (2023)
- [The Efficiency Spectrum of Large Language Models: An Algorithmic Survey](https://mlsanity.com/paper/2312.00678/) (2023)
- [MIMONets: Multiple-Input-Multiple-Output Neural Networks Exploiting Computation in Superposition](https://mlsanity.com/paper/2312.02829/) (2023)
- [Transformers Learn Low Sensitivity Functions: Investigations and Implications](https://mlsanity.com/paper/2403.06925/) (2024)
- [Homophone Disambiguation Reveals Patterns of Context Mixing in Speech Transformers](https://mlsanity.com/paper/2310.09925/) (2023)
- [Is Random Attention Sufficient for Sequence Modeling? Disentangling Trainable Components in the Transformer](https://mlsanity.com/paper/2506.01115/) (2025)
- [MERGE: Fast Private Text Generation](https://mlsanity.com/paper/2305.15769/) (2023)
- [Vocabulary-Defined Semantics: Latent Space Clustering for Improving In-Context Learning](https://mlsanity.com/paper/2401.16184/) (2024)
- [From Neurons to Neutrons: A Case Study in Interpretability](https://mlsanity.com/paper/2405.17425/) (2024)
- [Clustering and Alignment: Understanding the Training Dynamics in Modular Addition](https://mlsanity.com/paper/2408.09414/) (2024)
- [Optimization and Interpretability of Graph Attention Networks for Small Sparse Graph Structures in Automotive Applications](https://mlsanity.com/paper/2305.16196/) (2023)
- [Computational modeling of semantic change](https://mlsanity.com/paper/2304.06337/) (2023)
- [How Language Models Prioritize Contextual Grammatical Cues?](https://mlsanity.com/paper/2410.03447/) (2024)
- [Attention-guided Evidence Grounding for Spoken Question Answering](https://mlsanity.com/paper/2603.16292/) (2026)
- [When Can Attention Heads Be Statically Defined?](https://mlsanity.com/paper/2609.34650/) (2026)
- [Global Communication or Graph-Specific Memory?](https://mlsanity.com/paper/2610.05874/) (2026)

## Abstract (arXiv)

The attention mechanism is considered the backbone of the widely-used Transformer architecture. It contextualizes the input by computing input-specific attention matrices. We find that this mechanism, while powerful and elegant, is not as important as typically thought for pretrained language models. We introduce PAPA, a new probing method that replaces the input-dependent attention matrices with constant ones -- the average attention weights over multiple inputs. We use PAPA to analyze several established pretrained Transformers on six downstream tasks. We find that without any input-dependent attention, all models achieve competitive performance -- an average relative drop of only 8% from the probing baseline. Further, little or no performance drop is observed when replacing half of the input-dependent attention matrices with constant (input-independent) ones. Interestingly, we show that better-performing models lose more from applying our method than weaker models, suggesting that the utilization of the input-dependent attention mechanism might be a factor in their success. Our results motivate research on simpler alternatives to input-dependent attention, as well as on methods for better utilization of this mechanism in the Transformer architecture.