# 75 Languages, 1 Model: Parsing Universal Dependencies Universally

arXiv: https://arxiv.org/abs/1904.02099 · HTML: https://mlsanity.com/paper/1904.02099/

## Key facts

- Cited by 90 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Apr 2019.

## Built on by (most cited first, 30 of 90)

- [ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning](https://mlsanity.com/paper/2304.05613/) (2023)
- [Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.16039/) (2023)
- [Systematic Inequalities in Language Technology Performance across the World's Languages](https://mlsanity.com/paper/2110.06733/) (2021)
- [X-FACT: A New Benchmark Dataset for Multilingual Fact Checking](https://mlsanity.com/paper/2106.09248/) (2021)
- [Trankit: A Light-Weight Transformer-based Toolkit for Multilingual Natural Language Processing](https://mlsanity.com/paper/2101.03289/) (2021)
- [One Question Answering Model for Many Languages with Cross-lingual Dense Passage Retrieval](https://mlsanity.com/paper/2107.11976/) (2021)
- [Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation](https://mlsanity.com/paper/2203.09435/) (2022)
- [RobeCzech: Czech RoBERTa, a monolingual contextualized language representation model](https://mlsanity.com/paper/2105.11314/) (2021)
- [The Stem Cell Hypothesis: Dilemma behind Multi-Task Learning with Transformer Encoders](https://mlsanity.com/paper/2109.06939/) (2021)
- [Analyzing the Mono- and Cross-Lingual Pretraining Dynamics of Multilingual Language Models](https://mlsanity.com/paper/2205.11758/) (2022)
- [Zero-shot cross-lingual transfer language selection using linguistic similarity](https://mlsanity.com/paper/2301.13720/) (2023)
- [NorBench -- A Benchmark for Norwegian Language Models](https://mlsanity.com/paper/2305.03880/) (2023)
- [Attention Can Reflect Syntactic Structure (If You Let It)](https://mlsanity.com/paper/2101.10927/) (2021)
- [Specializing Multilingual Language Models: An Empirical Study](https://mlsanity.com/paper/2106.09063/) (2021)
- [Not all layers are equally as important: Every Layer Counts BERT](https://mlsanity.com/paper/2311.02265/) (2023)
- [Cross-lingual Visual Pre-training for Multimodal Machine Translation](https://mlsanity.com/paper/2101.10044/) (2021)
- [Subword Pooling Makes a Difference](https://mlsanity.com/paper/2102.10864/) (2021)
- [Protein Language Models and Structure Prediction: Connection and Progression](https://mlsanity.com/paper/2211.16742/) (2022)
- [One Model is All You Need: ByT5-Sanskrit, a Unified Model for Sanskrit NLP Tasks](https://mlsanity.com/paper/2409.13920/) (2024)
- [Pre-training Data Quality and Quantity for a Low-Resource Language: New Corpus and BERT Models for Maltese](https://mlsanity.com/paper/2205.10517/) (2022)
- [PhoNLP: A joint multi-task learning model for Vietnamese part-of-speech tagging, named entity recognition and dependency parsing](https://mlsanity.com/paper/2101.01476/) (2021)
- [Capitalization and Punctuation Restoration: a Survey](https://mlsanity.com/paper/2111.10746/) (2021)
- [Skill Extraction from Job Postings using Weak Supervision](https://mlsanity.com/paper/2209.08071/) (2022)
- [Meta-Learning for Fast Cross-Lingual Adaptation in Dependency Parsing](https://mlsanity.com/paper/2104.04736/) (2021)
- [Extending the Pre-Training of BLOOM for Improved Support of Traditional Chinese: Models, Methods and Results](https://mlsanity.com/paper/2303.04715/) (2023)
- [Taqyim: Evaluating Arabic NLP Tasks Using ChatGPT Models](https://mlsanity.com/paper/2306.16322/) (2023)
- [mCSQA: Multilingual Commonsense Reasoning Dataset with Unified Creation Strategy by Language Models and Humans](https://mlsanity.com/paper/2406.04215/) (2024)
- [A Little Pretraining Goes a Long Way: A Case Study on Dependency Parsing Task for Low-resource Morphologically Rich Languages](https://mlsanity.com/paper/2102.06551/) (2021)
- [Evaluating the Morphosyntactic Well-formedness of Generated Texts](https://mlsanity.com/paper/2103.16590/) (2021)
- [Genre as Weak Supervision for Cross-lingual Dependency Parsing](https://mlsanity.com/paper/2109.04733/) (2021)

## Abstract (arXiv)

We present UDify, a multilingual multi-task model capable of accurately predicting universal part-of-speech, morphological features, lemmas, and dependency trees simultaneously for all 124 Universal Dependencies treebanks across 75 languages. By leveraging a multilingual BERT self-attention model pretrained on 104 languages, we found that fine-tuning it on all datasets concatenated together with simple softmax classifiers for each UD task can result in state-of-the-art UPOS, UFeats, Lemmas, UAS, and LAS scores, without requiring any recurrent or language-specific components. We evaluate UDify for multilingual learning, showing that low-resource languages benefit the most from cross-linguistic annotations. We also evaluate for zero-shot learning, with results suggesting that multilingual training provides strong UD predictions even for languages that neither UDify nor BERT have ever been trained on. Code for UDify is available at https://github.com/hyperparticle/udify.