# Substructure Distribution Projection for Zero-Shot Cross-Lingual Dependency Parsing

arXiv: https://arxiv.org/abs/2110.08538 · HTML: https://mlsanity.com/paper/2110.08538/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 69 references that are in the map.
- First posted on arXiv 16 Oct 2021.

## Builds on (most cited first, 28 of 28)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [How multilingual is Multilingual BERT?](https://mlsanity.com/paper/1906.01502/) (2019)
- [Deep Biaffine Attention for Neural Dependency Parsing](https://mlsanity.com/paper/1611.01734/) (2016)
- [Universal Dependencies v2: An Evergrowing Multilingual Treebank Collection](https://mlsanity.com/paper/2004.10643/) (2020)
- [WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia](https://mlsanity.com/paper/1907.05791/) (2019)
- [SimAlign: High Quality Word Alignments without Parallel Training Data using Static and Contextualized Embeddings](https://mlsanity.com/paper/2004.08728/) (2020)
- [Cross-Lingual Alignment of Contextual Word Embeddings, with Applications to Zero-shot Dependency Parsing](https://mlsanity.com/paper/1902.09492/) (2019)
- [Cross-lingual Retrieval for Iterative Self-Supervised Training](https://mlsanity.com/paper/2006.09526/) (2020)
- [Efficient Second-Order TreeCRF for Neural Dependency Parsing](https://mlsanity.com/paper/2005.00975/) (2020)
- [Cross-Lingual BERT Transformation for Zero-Shot Dependency Parsing](https://mlsanity.com/paper/1909.06775/) (2019)
- [Single-/Multi-Source Cross-Lingual NER via Teacher-Student Learning on Unlabeled Data in Target Language](https://mlsanity.com/paper/2004.12440/) (2020)
- [Many Languages, One Parser](https://mlsanity.com/paper/1602.01595/) (2016)
- [Substructure Substitution: Structured Data Augmentation for NLP](https://mlsanity.com/paper/2101.00411/) (2021)
- [A Cross-Task Analysis of Text Span Representations](https://mlsanity.com/paper/2006.03866/) (2020)
- [Bilingual Lexicon Induction via Unsupervised Bitext Construction and Word Alignment](https://mlsanity.com/paper/2101.00148/) (2021)
- [Polyglot Contextual Representations Improve Crosslingual Transfer](https://mlsanity.com/paper/1902.09697/) (2019)
- [Cross-lingual Dependency Parsing with Unlabeled Auxiliary Languages](https://mlsanity.com/paper/1909.09265/) (2019)
- [Target Language-Aware Constrained Inference for Cross-lingual Dependency Parsing](https://mlsanity.com/paper/1909.01482/) (2019)
- [Cross-Lingual Dependency Parsing Using Code-Mixed TreeBank](https://mlsanity.com/paper/1909.02235/) (2019)
- [Cross-Lingual Syntactic Transfer with Limited Resources](https://mlsanity.com/paper/1610.06227/) (2016)
- [Low-Resource Syntactic Transfer with Unsupervised Source Reordering](https://mlsanity.com/paper/1903.05683/) (2019)
- [Cross-Lingual Syntactic Transfer through Unsupervised Adaptation of Invertible Projections](https://mlsanity.com/paper/1906.02656/) (2019)
- [PPT: Parsimonious Parser Transfer for Unsupervised Cross-Lingual Adaptation](https://mlsanity.com/paper/2101.11216/) (2021)

## Built on by (most cited first, 5 of 5)

- [Language Models are Multilingual Chain-of-Thought Reasoners](https://mlsanity.com/paper/2210.03057/) (2022)
- [Natural Language Processing RELIES on Linguistics](https://mlsanity.com/paper/2405.05966/) (2024)
- [Learning Language Structures through Grounding](https://mlsanity.com/paper/2406.09662/) (2024)
- [Incorporating Lexical and Syntactic Knowledge for Unsupervised Cross-Lingual Transfer](https://mlsanity.com/paper/2404.16627/) (2024)
- [Improving Low-Resource Cross-lingual Parsing with Expected Statistic Regularization](https://mlsanity.com/paper/2210.09428/) (2022)

## Abstract (arXiv)

We present substructure distribution projection (SubDP), a technique that projects a distribution over structures in one domain to another, by projecting substructure distributions separately. Models for the target domains can be then trained, using the projected distributions as soft silver labels. We evaluate SubDP on zero-shot cross-lingual dependency parsing, taking dependency arcs as substructures: we project the predicted dependency arc distributions in the source language(s) to target language(s), and train a target language parser to fit the resulting distributions. When an English treebank is the only annotation that involves human effort, SubDP achieves better unlabeled attachment score than all prior work on the Universal Dependencies v2.2 (Nivre et al., 2020) test set across eight diverse target languages, as well as the best labeled attachment score on six out of eight languages. In addition, SubDP improves zero-shot cross-lingual dependency parsing with very few (e.g., 50) supervised bitext pairs, across a broader range of target languages.