mlsanityData to 7 Oct 2026

The Fragility of Multi-Treebank Parsing Evaluation

arXiv 2209.06699 ↗First posted 14 Sep 2022cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Treebank selection for parsing evaluation and the spurious effects that might arise from a biased choice have not been explored in detail. This paper studies how evaluating on a single subset of treebanks can lead to weak conclusions. First, we take a few contrasting parsers, and run them on subsets of treebanks proposed in previous work, whose use was justified (or not) on criteria such as typology or data scarcity. Second, we run a large-scale version of this experiment, create vast amounts of random subsets of treebanks, and compare on them many parsers whose scores are available. The results show substantial variability across subsets and that although establishing guidelines for good treebank selection is hard, it is possible to detect potentially harmful strategies.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 20 of its 57 references that are in the map. First posted on arXiv 14 Sep 2022.

Builds on

20 of 57 references.

References · 20

  1. 2016Enriching Word Vectors with Subword Informationcited by 1,578
  2. 2022Challenges and Strategies in Cross-Cultural NLPcited by 216
  3. 2020We Need to Talk About Random Splitscited by 70
  4. 2019Universal Dependency Parsing from Scratchcited by 46
  5. 2018Stack-Pointer Networks for Dependency Parsingcited by 36
  6. 2019Viable Dependency Parsing as Sequence Labelingcited by 34
  7. 2018Towards Better UD Parsing: Deep Contextualized Word Embeddings, Ensemble, and Treebank Concatenationcited by 21
  8. 2019Left-to-Right Dependency Parsing with Pointer Networkscited by 21
  9. 2020Bracketing Encodings for 2-Planar Dependency Parsingcited by 15
  10. 2019Deep Contextualized Word Embeddings in Transition-Based and Graph-Based Dependency Parsing -- A Tale of Two Parsers Revisitedcited by 13
  11. 2020A Unifying Theory of Transition-based and Sequence Labeling Parsingcited by 12
  12. 2021Headed-Span-Based Projective Dependency Parsingcited by 11
8 more
  1. 2018An Investigation of the Interactions Between Pre-Trained Word Embeddings, Character Models and POS Tags in Dependency Parsingcited by 10
  2. 2021A Modest Pareto Optimisation Analysis of Dependency Parsers in 2021cited by 9
  3. 2019Global Greedy Dependency Parsingcited by 8
  4. 2021Not All Linearizations Are Equally Data-Hungry in Sequence Labeling Parsingcited by 7
  5. 2020Distilling Neural Networks for Greener and Faster Dependency Parsingcited by 5
  6. 2021Replicating and Extending "Because Their Treebanks Leak": Graph Isomorphism, Covariants, and Parser Performancecited by 5
  7. 2021A Falta de Pan, Buenas Son Tortas: The Efficacy of Predicted UPOS Tags for Low Resource UD Parsingcited by 5
  8. 2020Inherent Dependency Displacement Bias of Transition-Based Algorithmscited by 4

Built on by

Not cited yet.