# Distilling Neural Networks for Greener and Faster Dependency Parsing

arXiv: https://arxiv.org/abs/2006.00844 · HTML: https://mlsanity.com/paper/2006.00844/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Jun 2020.

## Built on by (most cited first, 5 of 5)

- [A Modest Pareto Optimisation Analysis of Dependency Parsers in 2021](https://mlsanity.com/paper/2106.04216/) (2021)
- [4 and 7-bit Labeling for Projective and Non-Projective Dependency Trees](https://mlsanity.com/paper/2310.14319/) (2023)
- [Not All Linearizations Are Equally Data-Hungry in Sequence Labeling Parsing](https://mlsanity.com/paper/2108.07556/) (2021)
- [Feature Alignment and Representation Transfer in Knowledge Distillation for Large Language Models](https://mlsanity.com/paper/2504.13825/) (2025)
- [The Fragility of Multi-Treebank Parsing Evaluation](https://mlsanity.com/paper/2209.06699/) (2022)

## Abstract (arXiv)

The carbon footprint of natural language processing research has been increasing in recent years due to its reliance on large and inefficient neural network implementations. Distillation is a network compression technique which attempts to impart knowledge from a large model to a smaller one. We use teacher-student distillation to improve the efficiency of the Biaffine dependency parser which obtains state-of-the-art performance with respect to accuracy and parsing speed (Dozat and Manning, 2017). When distilling to 20% of the original model's trainable parameters, we only observe an average decrease of ~1 point for both UAS and LAS across a number of diverse Universal Dependency treebanks while being 2.30x (1.19x) faster than the baseline model on CPU (GPU) at inference time. We also observe a small increase in performance when compressing to 80% for some treebanks. Finally, through distillation we attain a parser which is not only faster but also more accurate than the fastest modern parser on the Penn Treebank.