# The Fragility of Multi-Treebank Parsing Evaluation

arXiv: https://arxiv.org/abs/2209.06699 · HTML: https://mlsanity.com/paper/2209.06699/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 57 references that are in the map.
- First posted on arXiv 14 Sep 2022.

## Builds on (most cited first, 20 of 20)

- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Challenges and Strategies in Cross-Cultural NLP](https://mlsanity.com/paper/2203.10020/) (2022)
- [We Need to Talk About Random Splits](https://mlsanity.com/paper/2005.00636/) (2020)
- [Universal Dependency Parsing from Scratch](https://mlsanity.com/paper/1901.10457/) (2019)
- [Stack-Pointer Networks for Dependency Parsing](https://mlsanity.com/paper/1805.01087/) (2018)
- [Viable Dependency Parsing as Sequence Labeling](https://mlsanity.com/paper/1902.10505/) (2019)
- [Towards Better UD Parsing: Deep Contextualized Word Embeddings, Ensemble, and Treebank Concatenation](https://mlsanity.com/paper/1807.03121/) (2018)
- [Left-to-Right Dependency Parsing with Pointer Networks](https://mlsanity.com/paper/1903.08445/) (2019)
- [Bracketing Encodings for 2-Planar Dependency Parsing](https://mlsanity.com/paper/2011.00596/) (2020)
- [Deep Contextualized Word Embeddings in Transition-Based and Graph-Based Dependency Parsing -- A Tale of Two Parsers Revisited](https://mlsanity.com/paper/1908.07397/) (2019)
- [A Unifying Theory of Transition-based and Sequence Labeling Parsing](https://mlsanity.com/paper/2011.00584/) (2020)
- [Headed-Span-Based Projective Dependency Parsing](https://mlsanity.com/paper/2108.04750/) (2021)
- [An Investigation of the Interactions Between Pre-Trained Word Embeddings, Character Models and POS Tags in Dependency Parsing](https://mlsanity.com/paper/1808.09060/) (2018)
- [A Modest Pareto Optimisation Analysis of Dependency Parsers in 2021](https://mlsanity.com/paper/2106.04216/) (2021)
- [Global Greedy Dependency Parsing](https://mlsanity.com/paper/1911.08673/) (2019)
- [Not All Linearizations Are Equally Data-Hungry in Sequence Labeling Parsing](https://mlsanity.com/paper/2108.07556/) (2021)
- [Distilling Neural Networks for Greener and Faster Dependency Parsing](https://mlsanity.com/paper/2006.00844/) (2020)
- [Replicating and Extending "Because Their Treebanks Leak": Graph Isomorphism, Covariants, and Parser Performance](https://mlsanity.com/paper/2106.00352/) (2021)
- [A Falta de Pan, Buenas Son Tortas: The Efficacy of Predicted UPOS Tags for Low Resource UD Parsing](https://mlsanity.com/paper/2106.04222/) (2021)
- [Inherent Dependency Displacement Bias of Transition-Based Algorithms](https://mlsanity.com/paper/2003.14282/) (2020)

## Abstract (arXiv)

Treebank selection for parsing evaluation and the spurious effects that might arise from a biased choice have not been explored in detail. This paper studies how evaluating on a single subset of treebanks can lead to weak conclusions. First, we take a few contrasting parsers, and run them on subsets of treebanks proposed in previous work, whose use was justified (or not) on criteria such as typology or data scarcity. Second, we run a large-scale version of this experiment, create vast amounts of random subsets of treebanks, and compare on them many parsers whose scores are available. The results show substantial variability across subsets and that although establishing guidelines for good treebank selection is hard, it is possible to detect potentially harmful strategies.