# Universal Dependencies according to BERT: both more specific and more general

arXiv: https://arxiv.org/abs/2004.14620 · HTML: https://mlsanity.com/paper/2004.14620/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Apr 2020.

## Built on by (most cited first, 11 of 11)

- [State-of-the-art generalisation research in NLP: A taxonomy and review](https://mlsanity.com/paper/2210.03050/) (2022)
- [A Primer on Pretrained Multilingual Language Models](https://mlsanity.com/paper/2107.00676/) (2021)
- [Mini-Model Adaptation: Efficiently Extending Pretrained Models to New Languages via Aligned Shallow Training](https://mlsanity.com/paper/2212.10503/) (2022)
- [Attention Can Reflect Syntactic Structure (If You Let It)](https://mlsanity.com/paper/2101.10927/) (2021)
- [Linguistic Interpretability of Transformer-based Language Models: a systematic review](https://mlsanity.com/paper/2504.08001/) (2025)
- [On the Evolution of Syntactic Information Encoded by BERT's Contextualized Representations](https://mlsanity.com/paper/2101.11492/) (2021)
- [Multilingual Transformer Encoders: a Word-Level Task-Agnostic Evaluation](https://mlsanity.com/paper/2207.09076/) (2022)
- [Syntactic Substitutability as Unsupervised Dependency Syntax](https://mlsanity.com/paper/2211.16031/) (2022)
- [Inducing Systematicity in Transformers by Attending to Structurally Quantized Embeddings](https://mlsanity.com/paper/2402.06492/) (2024)
- [The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Models](https://mlsanity.com/paper/2601.19926/) (2026)
- [Dynamic Syntax Mapping: A New Approach to Unsupervised Syntax Parsing](https://mlsanity.com/paper/2312.14966/) (2023)

## Abstract (arXiv)

This work focuses on analyzing the form and extent of syntactic abstraction captured by BERT by extracting labeled dependency trees from self-attentions. Previous work showed that individual BERT heads tend to encode particular dependency relation types. We extend these findings by explicitly comparing BERT relations to Universal Dependencies (UD) annotations, showing that they often do not match one-to-one. We suggest a method for relation identification and syntactic tree construction. Our approach produces significantly more consistent dependency trees than previous work, showing that it better explains the syntactic abstractions in BERT. At the same time, it can be successfully applied with only a minimal amount of supervision and generalizes well across languages.