# Heads-up! Unsupervised Constituency Parsing via Self-Attention Heads

arXiv: https://arxiv.org/abs/2010.09517 · HTML: https://mlsanity.com/paper/2010.09517/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Oct 2020.

## Built on by (most cited first, 12 of 12)

- [It's All in the Heads: Using Attention Heads as a Baseline for Cross-Lingual Transfer in Commonsense Reasoning](https://mlsanity.com/paper/2106.12066/) (2021)
- [Proton: Probing Schema Linking Information from Pre-trained Language Models for Text-to-SQL Parsing](https://mlsanity.com/paper/2206.14017/) (2022)
- [Neural Bi-Lexicalized PCFG Induction](https://mlsanity.com/paper/2105.15021/) (2021)
- [Contextual Distortion Reveals Constituency: Masked Language Models are Implicit Parsers](https://mlsanity.com/paper/2306.00645/) (2023)
- [Co-training an Unsupervised Constituency Parser with Weak Supervision](https://mlsanity.com/paper/2110.02283/) (2021)
- [Forming Trees with Treeformers](https://mlsanity.com/paper/2207.06960/) (2022)
- [Approximating CKY with Transformers](https://mlsanity.com/paper/2305.02386/) (2023)
- [Revisiting the Practical Effectiveness of Constituency Parse Extraction from Pre-trained Language Models](https://mlsanity.com/paper/2211.00479/) (2022)
- [Focused Contrastive Training for Test-based Constituency Analysis](https://mlsanity.com/paper/2109.15159/) (2021)
- [Unsupervised and Few-shot Parsing from Pretrained Language Models](https://mlsanity.com/paper/2206.04980/) (2022)
- [On Eliciting Syntax from Language Models via Hashing](https://mlsanity.com/paper/2410.04074/) (2024)
- [Understanding Syntactic Generalization in Structure-inducing Language Models](https://mlsanity.com/paper/2508.07969/) (2025)

## Abstract (arXiv)

Transformer-based pre-trained language models (PLMs) have dramatically improved the state of the art in NLP across many tasks. This has led to substantial interest in analyzing the syntactic knowledge PLMs learn. Previous approaches to this question have been limited, mostly using test suites or probes. Here, we propose a novel fully unsupervised parsing approach that extracts constituency trees from PLM attention heads. We rank transformer attention heads based on their inherent properties, and create an ensemble of high-ranking heads to produce the final tree. Our method is adaptable to low-resource languages, as it does not rely on development sets, which can be expensive to annotate. Our experiments show that the proposed method often outperform existing approaches if there is no development set present. Our unsupervised parser can also be used as a tool to analyze the grammars PLMs learn implicitly. For this, we use the parse trees induced by our method to train a neural PCFG and compare it to a grammar derived from a human-annotated treebank.