# Minor changes make a difference: a case study on the consistency of UD-based dependency parsers

arXiv: https://arxiv.org/abs/2111.15413 · HTML: https://mlsanity.com/paper/2111.15413/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 12 references that are in the map.
- First posted on arXiv 30 Nov 2021.

## Builds on (most cited first, 5 of 5)

- [Stanza: A Python Natural Language Processing Toolkit for Many Human Languages](https://mlsanity.com/paper/2003.07082/) (2020)
- [All Roads Lead to UD: Converting Stanford and Penn Parses to English Universal Dependencies with Multilayer Annotations](https://mlsanity.com/paper/1909.00522/) (2019)
- [Automatic Extraction of Rules Governing Morphological Agreement](https://mlsanity.com/paper/2010.01160/) (2020)
- [Evaluating the Morphosyntactic Well-formedness of Generated Texts](https://mlsanity.com/paper/2103.16590/) (2021)
- [Quinductor: a multilingual data-driven method for generating reading-comprehension questions using Universal Dependencies](https://mlsanity.com/paper/2103.10121/) (2021)

## Built on by (most cited first, 2 of 2)

- [Are UD Treebanks Getting More Consistent? A Report Card for English UD](https://mlsanity.com/paper/2302.00636/) (2023)
- [Automatically generating question-answer pairs for assessing basic reading comprehension in Swedish](https://mlsanity.com/paper/2211.15568/) (2022)

## Abstract (arXiv)

Many downstream applications are using dependency trees, and are thus relying on dependency parsers producing correct, or at least consistent, output. However, dependency parsers are trained using machine learning, and are therefore susceptible to unwanted inconsistencies due to biases in the training data. This paper explores the effects of such biases in four languages - English, Swedish, Russian, and Ukrainian - though an experiment where we study the effect of replacing numerals in sentences. We show that such seemingly insignificant changes in the input can cause large differences in the output, and suggest that data augmentation can remedy the problems.