# Inducing Syntactic Trees from BERT Representations

arXiv: https://arxiv.org/abs/1906.11511 · HTML: https://mlsanity.com/paper/1906.11511/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Jun 2019.

## Built on by (most cited first, 11 of 11)

- [Pre-Trained Models: Past, Present and Future](https://mlsanity.com/paper/2106.07139/) (2021)
- [Attention Can Reflect Syntactic Structure (If You Let It)](https://mlsanity.com/paper/2101.10927/) (2021)
- [Interpreting Deep Learning Models in Natural Language Processing: A Review](https://mlsanity.com/paper/2110.10470/) (2021)
- [Shaking Syntactic Trees on the Sesame Street: Multilingual Probing with Controllable Perturbations](https://mlsanity.com/paper/2109.14017/) (2021)
- [What does Transformer learn about source code?](https://mlsanity.com/paper/2207.08466/) (2022)
- [Probing Script Knowledge from Pre-Trained Models](https://mlsanity.com/paper/2204.10176/) (2022)
- [Topic Aware Probing: From Sentence Length Prediction to Idiom Identification how reliant are Neural Language Models on Topic?](https://mlsanity.com/paper/2403.02009/) (2024)
- [Construction Grammar and Language Models](https://mlsanity.com/paper/2308.13315/) (2023)
- [Understanding the Role of Input Token Characters in Language Models: How Does Information Loss Affect Performance?](https://mlsanity.com/paper/2310.17271/) (2023)
- [Revisiting the Practical Effectiveness of Constituency Parse Extraction from Pre-trained Language Models](https://mlsanity.com/paper/2211.00479/) (2022)
- [The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Models](https://mlsanity.com/paper/2601.19926/) (2026)

## Abstract (arXiv)

We use the English model of BERT and explore how a deletion of one word in a sentence changes representations of other words. Our hypothesis is that removing a reducible word (e.g. an adjective) does not affect the representation of other words so much as removing e.g. the main verb, which makes the sentence ungrammatical and of "high surprise" for the language model. We estimate reducibilities of individual words and also of longer continuous phrases (word n-grams), study their syntax-related properties, and then also use them to induce full dependency trees.