# Parsing with Pretrained Language Models, Multiple Datasets, and Dataset Embeddings

arXiv: https://arxiv.org/abs/2112.03625 · HTML: https://mlsanity.com/paper/2112.03625/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 15 references that are in the map.
- First posted on arXiv 7 Dec 2021.

## Builds on (most cited first, 12 of 12)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Deep Biaffine Attention for Neural Dependency Parsing](https://mlsanity.com/paper/1611.01734/) (2016)
- [Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT](https://mlsanity.com/paper/1904.09077/) (2019)
- [Universal Dependencies v2: An Evergrowing Multilingual Treebank Collection](https://mlsanity.com/paper/2004.10643/) (2020)
- [75 Languages, 1 Model: Parsing Universal Dependencies Universally](https://mlsanity.com/paper/1904.02099/) (2019)
- [Massive Choice, Ample Tasks (MaChAmp): A Toolkit for Multi-task Learning in NLP](https://mlsanity.com/paper/2005.14672/) (2020)
- [Many Languages, One Parser](https://mlsanity.com/paper/1602.01595/) (2016)
- [82 Treebanks, 34 Models: Universal Dependency Parsing with Multi-Treebank Models](https://mlsanity.com/paper/1809.02237/) (2018)
- [Parser Training with Heterogeneous Treebanks](https://mlsanity.com/paper/1805.05089/) (2018)
- [One model, two languages: training bilingual parsers with harmonized treebanks](https://mlsanity.com/paper/1507.08449/) (2015)
- [Treebank Embedding Vectors for Out-of-domain Dependency Parsing](https://mlsanity.com/paper/2005.00800/) (2020)

## Built on by (most cited first, 1 of 1)

- [How to Encode Domain Information in Relation Classification](https://mlsanity.com/paper/2404.13760/) (2024)

## Abstract (arXiv)

With an increase of dataset availability, the potential for learning from a variety of data sources has increased. One particular method to improve learning from multiple data sources is to embed the data source during training. This allows the model to learn generalizable features as well as distinguishing features between datasets. However, these dataset embeddings have mostly been used before contextualized transformer-based embeddings were introduced in the field of Natural Language Processing. In this work, we compare two methods to embed datasets in a transformer-based multilingual dependency parser, and perform an extensive evaluation. We show that: 1) embedding the dataset is still beneficial with these models 2) performance increases are highest when embedding the dataset at the encoder level 3) unsurprisingly, we confirm that performance increases are highest for small datasets and datasets with a low baseline score. 4) we show that training on the combination of all datasets performs similarly to designing smaller clusters based on language-relatedness.