# Improving Low-Resource Cross-lingual Parsing with Expected Statistic Regularization

arXiv: https://arxiv.org/abs/2210.09428 · HTML: https://mlsanity.com/paper/2210.09428/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 53 references that are in the map.
- First posted on arXiv 17 Oct 2022.

## Builds on (most cited first, 26 of 26)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Deep Biaffine Attention for Neural Dependency Parsing](https://mlsanity.com/paper/1611.01734/) (2016)
- [Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT](https://mlsanity.com/paper/1904.09077/) (2019)
- [Data Programming: Creating Large Training Sets, Quickly](https://mlsanity.com/paper/1605.07723/) (2016)
- [Choosing Transfer Languages for Cross-Lingual Learning](https://mlsanity.com/paper/1905.12688/) (2019)
- [Yara Parser: A Fast and Accurate Dependency Parser](https://mlsanity.com/paper/1503.06733/) (2015)
- [75 Languages, 1 Model: Parsing Universal Dependencies Universally](https://mlsanity.com/paper/1904.02099/) (2019)
- [UDapter: Language Adaptation for Truly Universal Dependency Parsing](https://mlsanity.com/paper/2004.14327/) (2020)
- [Gradient Estimation with Stochastic Softmax Tricks](https://mlsanity.com/paper/2006.08063/) (2020)
- [Many Languages, One Parser](https://mlsanity.com/paper/1602.01595/) (2016)
- [Multiple Instance Learning Networks for Fine-Grained Sentiment Analysis](https://mlsanity.com/paper/1711.09645/) (2017)
- [Zero-shot Dependency Parsing with Pre-trained Multilingual Sentence Representations](https://mlsanity.com/paper/1910.05479/) (2019)
- [Deep Contextualized Word Embeddings in Transition-Based and Graph-Based Dependency Parsing -- A Tale of Two Parsers Revisited](https://mlsanity.com/paper/1908.07397/) (2019)
- [Cross-lingual Dependency Parsing with Unlabeled Auxiliary Languages](https://mlsanity.com/paper/1909.09265/) (2019)
- [Target Language-Aware Constrained Inference for Cross-lingual Dependency Parsing](https://mlsanity.com/paper/1909.01482/) (2019)
- [Leveraging Just a Few Keywords for Fine-Grained Aspect Detection Through Weakly Supervised Co-Training](https://mlsanity.com/paper/1909.00415/) (2019)
- [Cross-Lingual Syntactic Transfer with Limited Resources](https://mlsanity.com/paper/1610.06227/) (2016)
- [Low-Resource Syntactic Transfer with Unsupervised Source Reordering](https://mlsanity.com/paper/1903.05683/) (2019)
- [Cross-Lingual Syntactic Transfer through Unsupervised Adaptation of Invertible Projections](https://mlsanity.com/paper/1906.02656/) (2019)
- [PPT: Parsimonious Parser Transfer for Unsupervised Cross-Lingual Adaptation](https://mlsanity.com/paper/2101.11216/) (2021)
- [Substructure Distribution Projection for Zero-Shot Cross-Lingual Dependency Parsing](https://mlsanity.com/paper/2110.08538/) (2021)
- [Weakly Supervised Attention Networks for Fine-Grained Opinion Mining and Public Health](https://mlsanity.com/paper/1910.00054/) (2019)
- [Transfer Learning Between Related Tasks Using Expected Label Proportions](https://mlsanity.com/paper/1909.00430/) (2019)

## Abstract (arXiv)

We present Expected Statistic Regularization (ESR), a novel regularization technique that utilizes low-order multi-task structural statistics to shape model distributions for semi-supervised learning on low-resource datasets. We study ESR in the context of cross-lingual transfer for syntactic analysis (POS tagging and labeled dependency parsing) and present several classes of low-order statistic functions that bear on model behavior. Experimentally, we evaluate the proposed statistics with ESR for unsupervised transfer on 5 diverse target languages and show that all statistics, when estimated accurately, yield improvements to both POS and LAS, with the best statistic improving POS by +7.0 and LAS by +8.5 on average. We also present semi-supervised transfer and learning curve experiments that show ESR provides significant gains over strong cross-lingual-transfer-plus-fine-tuning baselines for modest amounts of label data. These results indicate that ESR is a promising and complementary approach to model-transfer approaches for cross-lingual parsing.