# The heads hypothesis: A unifying statistical approach towards understanding multi-headed attention in BERT arXiv: https://arxiv.org/abs/2101.09115 ยท HTML: https://mlsanity.com/paper/2101.09115/ ## Key facts - Cited by 15 papers in the map (data through 14 Oct 2026). - Builds on 27 of its 33 references that are in the map. - First posted on arXiv 22 Jan 2021. ## Builds on (most cited first, 27 of 27) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018) - [What Does BERT Look At? An Analysis of BERT's Attention](https://mlsanity.com/paper/1906.04341/) (2019) - [Attention is not Explanation](https://mlsanity.com/paper/1902.10186/) (2019) - [Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned](https://mlsanity.com/paper/1905.09418/) (2019) - [Are Sixteen Heads Really Better than One?](https://mlsanity.com/paper/1905.10650/) (2019) - [A Primer in BERTology: What we know about how BERT works](https://mlsanity.com/paper/2002.12327/) (2020) - [How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings](https://mlsanity.com/paper/1909.00512/) (2019) - [Attention is not not Explanation](https://mlsanity.com/paper/1908.04626/) (2019) - [What do you learn from context? Probing for sentence structure in contextualized word representations](https://mlsanity.com/paper/1905.06316/) (2019) - [Linguistic Knowledge and Transferability of Contextual Representations](https://mlsanity.com/paper/1903.08855/) (2019) - [What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models](https://mlsanity.com/paper/1907.13528/) (2019) - [Revealing the Dark Secrets of BERT](https://mlsanity.com/paper/1908.08593/) (2019) - [Pay Less Attention with Lightweight and Dynamic Convolutions](https://mlsanity.com/paper/1901.10430/) (2019) - [Assessing BERT's Syntactic Abilities](https://mlsanity.com/paper/1901.05287/) (2019) - [Adaptively Sparse Transformers](https://mlsanity.com/paper/1909.00015/) (2019) - [Open Sesame: Getting Inside BERT's Linguistic Knowledge](https://mlsanity.com/paper/1906.01698/) (2019) - [Do Attention Heads in BERT Track Syntactic Dependencies?](https://mlsanity.com/paper/1911.12246/) (2019) - [Visualizing and Understanding the Effectiveness of BERT](https://mlsanity.com/paper/1908.05620/) (2019) - [Does BERT Make Any Sense? Interpretable Word Sense Disambiguation with Contextualized Embeddings](https://mlsanity.com/paper/1909.10430/) (2019) - [Modeling Localness for Self-Attention Networks](https://mlsanity.com/paper/1810.10182/) (2018) - [Fixed Encoder Self-Attention Patterns in Transformer-Based Machine Translation](https://mlsanity.com/paper/2002.10260/) (2020) - [Convolutional Self-Attention Networks](https://mlsanity.com/paper/1904.03107/) (2019) - [Complexity of Word Collocation Networks: A Preliminary Structural Analysis](https://mlsanity.com/paper/1310.5111/) (2013) - [What do you mean, BERT? Assessing BERT as a Distributional Semantics Model](https://mlsanity.com/paper/1911.05758/) (2019) - [Not All Claims are Created Equal: Choosing the Right Statistical Approach to Assess Hypotheses](https://mlsanity.com/paper/1911.03850/) (2019) ## Built on by (most cited first, 15 of 15) - [Attention Heads of Large Language Models: A Survey](https://mlsanity.com/paper/2409.03752/) (2024) - [Towards Building ASR Systems for the Next Billion Users](https://mlsanity.com/paper/2111.03945/) (2021) - [Acceptability Judgements via Examining the Topology of Attention Maps](https://mlsanity.com/paper/2205.09630/) (2022) - [Robust AI-Generated Text Detection by Restricted Embeddings](https://mlsanity.com/paper/2410.08113/) (2024) - [Linguistic Interpretability of Transformer-based Language Models: a systematic review](https://mlsanity.com/paper/2504.08001/) (2025) - [Improving Interpretability and Robustness for the Detection of AI-Generated Images](https://mlsanity.com/paper/2406.15035/) (2024) - [On the Prunability of Attention Heads in Multilingual BERT](https://mlsanity.com/paper/2109.12683/) (2021) - [Analysis of Argument Structure Constructions in the Large Language Model BERT](https://mlsanity.com/paper/2408.04270/) (2024) - [A Novel Perspective to Look At Attention: Bi-level Attention-based Explainable Topic Modeling for News Classification](https://mlsanity.com/paper/2203.07216/) (2022) - [Input-specific Attention Subnetworks for Adversarial Detection](https://mlsanity.com/paper/2203.12298/) (2022) - [AttentionMix: Data augmentation method that relies on BERT attention mechanism](https://mlsanity.com/paper/2309.11104/) (2023) - [Listening to the Wise Few: Query-Key Alignment Unlocks Latent Correct Answers in Large Language Models](https://mlsanity.com/paper/2410.02343/) (2024) - [The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Models](https://mlsanity.com/paper/2601.19926/) (2026) - [Explainability of Text Processing and Retrieval Methods: A Survey](https://mlsanity.com/paper/2212.07126/) (2022) - [AxBERT: An Interpretable Chinese Spelling Correction Method Driven by Associative Knowledge Network](https://mlsanity.com/paper/2503.02255/) (2025) ## Abstract (arXiv) Multi-headed attention heads are a mainstay in transformer-based models. Different methods have been proposed to classify the role of each attention head based on the relations between tokens which have high pair-wise attention. These roles include syntactic (tokens with some syntactic relation), local (nearby tokens), block (tokens in the same sentence) and delimiter (the special [CLS], [SEP] tokens). There are two main challenges with existing methods for classification: (a) there are no standard scores across studies or across functional roles, and (b) these scores are often average quantities measured across sentences without capturing statistical significance. In this work, we formalize a simple yet effective score that generalizes to all the roles of attention heads and employs hypothesis testing on this score for robust inference. This provides us the right lens to systematically analyze attention heads and confidently comment on many commonly posed questions on analyzing the BERT model. In particular, we comment on the co-location of multiple functional roles in the same attention head, the distribution of attention heads across layers, and effect of fine-tuning for specific NLP tasks on these functional roles.