# Replicating and Extending "Because Their Treebanks Leak": Graph Isomorphism, Covariants, and Parser Performance

arXiv: https://arxiv.org/abs/2106.00352 · HTML: https://mlsanity.com/paper/2106.00352/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 29 references that are in the map.
- First posted on arXiv 1 Jun 2021.

## Builds on (most cited first, 6 of 6)

- [Deep Biaffine Attention for Neural Dependency Parsing](https://mlsanity.com/paper/1611.01734/) (2016)
- [Viable Dependency Parsing as Sequence Labeling](https://mlsanity.com/paper/1902.10505/) (2019)
- [Dissecting Span Identification Tasks with Performance Prediction](https://mlsanity.com/paper/2010.02587/) (2020)
- [Efficient EUD Parsing](https://mlsanity.com/paper/2006.00838/) (2020)
- [The risks of mixing dependency lengths from sequences of different length](https://mlsanity.com/paper/1304.3841/) (2013)
- [Inherent Dependency Displacement Bias of Transition-Based Algorithms](https://mlsanity.com/paper/2003.14282/) (2020)

## Built on by (most cited first, 5 of 5)

- [Investigating data partitioning strategies for crosslinguistic low-resource ASR evaluation](https://mlsanity.com/paper/2208.12888/) (2022)
- [Data-driven Parsing Evaluation for Child-Parent Interactions](https://mlsanity.com/paper/2209.13778/) (2022)
- [Better Benchmarking LLMs for Zero-Shot Dependency Parsing](https://mlsanity.com/paper/2502.20866/) (2025)
- [The Fragility of Multi-Treebank Parsing Evaluation](https://mlsanity.com/paper/2209.06699/) (2022)
- [The Impact of Edge Displacement Vaserstein Distance on UD Parsing Performance](https://mlsanity.com/paper/2209.07139/) (2022)

## Abstract (arXiv)

Sogaard (2020) obtained results suggesting the fraction of trees occurring in the test data isomorphic to trees in the training set accounts for a non-trivial variation in parser performance. Similar to other statistical analyses in NLP, the results were based on evaluating linear regressions. However, the study had methodological issues and was undertaken using a small sample size leading to unreliable results. We present a replication study in which we also bin sentences by length and find that only a small subset of sentences vary in performance with respect to graph isomorphism. Further, the correlation observed between parser performance and graph isomorphism in the wild disappears when controlling for covariants. However, in a controlled experiment, where covariants are kept fixed, we do observe a strong correlation. We suggest that conclusions drawn from statistical analyses like this need to be tempered and that controlled experiments can complement them by more readily teasing factors apart.