mlsanityData to 7 Oct 2026

Have Attention Heads in BERT Learned Constituency Grammar?

arXiv 2102.07926 ↗First posted 16 Feb 2021cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

With the success of pre-trained language models in recent years, more and more researchers focus on opening the "black box" of these models. Following this interest, we carry out a qualitative and quantitative analysis of constituency grammar in attention heads of BERT and RoBERTa. We employ the syntactic distance method to extract implicit constituency grammar from the attention weights of each head. Our results show that there exist heads that can induce some grammar types much better than baselines, suggesting that some heads act as a proxy for constituency grammar. We also analyze how attention heads' constituency grammar inducing (CGI) ability changes after fine-tuning with two kinds of tasks, including sentence meaning similarity (SMS) tasks and natural language inference (NLI) tasks. Our results suggest that SMS tasks decrease the average CGI ability of upper layers, while NLI tasks increase it. Lastly, we investigate the connections between CGI ability and natural language understanding ability on QQP and MNLI tasks.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 20 of its 26 references that are in the map. First posted on arXiv 16 Feb 2021.

Builds on

20 of 26 references.

References · 20

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  6. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  7. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  8. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  9. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  10. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  11. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  12. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
8 more
  1. 2019Are Sixteen Heads Really Better than One?cited by 803
  2. 2019Revealing the Dark Secrets of BERTcited by 300
  3. 2019Assessing BERT's Syntactic Abilitiescited by 191
  4. 2019Do Attention Heads in BERT Track Syntactic Dependencies?cited by 87
  5. 2020Fast and Accurate Neural CRF Constituency Parsingcited by 50
  6. 2020Are Pre-trained Language Models Aware of Phrases? Simple but Strong Baselines for Grammar Inductioncited by 47
  7. 2020Is Supervised Syntactic Parsing Beneficial for Language Understanding? An Empirical Investigationcited by 34
  8. 2018Straight to the Tree: Constituency Parsing with Neural Syntactic Distancecited by 32

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 0; 2024: 1; 2025: 1; 2026: 1.

5 papers.

  1. 2022Acceptability Judgements via Examining the Topology of Attention Mapscited by 19
  2. 2025Linguistic Interpretability of Transformer-based Language Models: a systematic reviewcited by 7
  3. 2024Are there identifiable structural parts in the sentence embedding whole?cited by 6
  4. 2022Improving word mover's distance by leveraging self-attention matrixcited by 3
  5. 2026The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Modelscited by 1