# Distilling an Ensemble of Greedy Dependency Parsers into One MST Parser

arXiv: https://arxiv.org/abs/1609.07561 · HTML: https://mlsanity.com/paper/1609.07561/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Sep 2016.

## Built on by (most cited first, 11 of 11)

- [Autoregressive Knowledge Distillation through Imitation Learning](https://mlsanity.com/paper/2009.07253/) (2020)
- [Follow the Wisdom of the Crowd: Effective Text Generation via Minimum Bayes Risk Decoding](https://mlsanity.com/paper/2211.07634/) (2022)
- [Maximum Bayes Smatch Ensemble Distillation for AMR Parsing](https://mlsanity.com/paper/2112.07790/) (2021)
- [Ensembling Graph Predictions for AMR Parsing](https://mlsanity.com/paper/2110.09131/) (2021)
- [R-U-SURE? Uncertainty-Aware Code Suggestions By Maximizing Utility Across Random User Intents](https://mlsanity.com/paper/2303.00732/) (2023)
- [A Modest Pareto Optimisation Analysis of Dependency Parsers in 2021](https://mlsanity.com/paper/2106.04216/) (2021)
- [Multilingual AMR Parsing with Noisy Knowledge Distillation](https://mlsanity.com/paper/2109.15196/) (2021)
- [Transformers as Graph-to-Graph Models](https://mlsanity.com/paper/2310.17936/) (2023)
- [Learning Energy-Based Approximate Inference Networks for Structured Applications in NLP](https://mlsanity.com/paper/2108.12522/) (2021)
- [CPTAM: Constituency Parse Tree Aggregation Method](https://mlsanity.com/paper/2201.07905/) (2022)
- [Empirical Analysis for Unsupervised Universal Dependency Parse Tree Aggregation](https://mlsanity.com/paper/2403.19183/) (2024)

## Abstract (arXiv)

We introduce two first-order graph-based dependency parsers achieving a new state of the art. The first is a consensus parser built from an ensemble of independently trained greedy LSTM transition-based parsers with different random initializations. We cast this approach as minimum Bayes risk decoding (under the Hamming cost) and argue that weaker consensus within the ensemble is a useful signal of difficulty or ambiguity. The second parser is a "distillation" of the ensemble into a single model. We train the distillation parser using a structured hinge loss objective with a novel cost that incorporates ensemble uncertainty estimates for each possible attachment, thereby avoiding the intractable cross-entropy computations required by applying standard distillation objectives to problems with structured outputs. The first-order distillation parser matches or surpasses the state of the art on English, Chinese, and German.