# Applying Occam's Razor to Transformer-Based Dependency Parsing: What Works, What Doesn't, and What is Really Necessary

arXiv: https://arxiv.org/abs/2010.12699 · HTML: https://mlsanity.com/paper/2010.12699/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Oct 2020.

## Built on by (most cited first, 8 of 8)

- [A Survey of Methods for Addressing Class Imbalance in Deep-Learning Based Natural Language Processing](https://mlsanity.com/paper/2210.04675/) (2022)
- [Dependency Annotation of Ottoman Turkish with Multilingual BERT](https://mlsanity.com/paper/2402.14743/) (2024)
- [BoschAI @ Causal News Corpus 2023: Robust Cause-Effect Span Extraction using Multi-Layer Sequence Tagging and Data Augmentation](https://mlsanity.com/paper/2312.06338/) (2023)
- [Emergence of Phonemic, Syntactic, and Semantic Representations in Artificial Neural Networks](https://mlsanity.com/paper/2601.18617/) (2026)
- [MuLMS: A Multi-Layer Annotated Text Corpus for Information Extraction in the Materials Science Domain](https://mlsanity.com/paper/2310.15569/) (2023)
- [TuGeBiC: A Turkish German Bilingual Code-Switching Corpus](https://mlsanity.com/paper/2205.00868/) (2022)
- [Building Foundations for Natural Language Processing of Historical Turkish: Resources and Models](https://mlsanity.com/paper/2501.04828/) (2025)
- [Dependency Parsing Across the Resource Spectrum: Evaluating Architectures on High and Low-Resource Languages](https://mlsanity.com/paper/2605.02608/) (2026)

## Abstract (arXiv)

The introduction of pre-trained transformer-based contextualized word embeddings has led to considerable improvements in the accuracy of graph-based parsers for frameworks such as Universal Dependencies (UD). However, previous works differ in various dimensions, including their choice of pre-trained language models and whether they use LSTM layers. With the aims of disentangling the effects of these choices and identifying a simple yet widely applicable architecture, we introduce STEPS, a new modular graph-based dependency parser. Using STEPS, we perform a series of analyses on the UD corpora of a diverse set of languages. We find that the choice of pre-trained embeddings has by far the greatest impact on parser performance and identify XLM-R as a robust choice across the languages in our study. Adding LSTM layers provides no benefits when using transformer-based embeddings. A multi-task training setup outputting additional UD features may contort results. Taking these insights together, we propose a simple but widely applicable parser architecture and configuration, achieving new state-of-the-art results (in terms of LAS) for 10 out of 12 diverse languages.