# Everything Is All It Takes: A Multipronged Strategy for Zero-Shot Cross-Lingual Information Extraction

arXiv: https://arxiv.org/abs/2109.06798 · HTML: https://mlsanity.com/paper/2109.06798/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 79 references that are in the map.
- First posted on arXiv 14 Sep 2021.

## Builds on (most cited first, 30 of 41)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization](https://mlsanity.com/paper/2003.11080/) (2020)
- [Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation](https://mlsanity.com/paper/1611.04558/) (2016)
- [AllenNLP: A Deep Semantic Natural Language Processing Platform](https://mlsanity.com/paper/1803.07640/) (2018)
- [Deep Biaffine Attention for Neural Dependency Parsing](https://mlsanity.com/paper/1611.01734/) (2016)
- [Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT](https://mlsanity.com/paper/1904.09077/) (2019)
- [WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia](https://mlsanity.com/paper/1907.05791/) (2019)
- [Are All Languages Created Equal in Multilingual BERT?](https://mlsanity.com/paper/2005.09093/) (2020)
- [Word Alignment by Fine-tuning Embeddings on Parallel Corpora](https://mlsanity.com/paper/2101.08231/) (2021)
- [Emerging Cross-lingual Structure in Pretrained Language Models](https://mlsanity.com/paper/1911.01464/) (2019)
- [XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation](https://mlsanity.com/paper/2104.07412/) (2021)
- [SimAlign: High Quality Word Alignments without Parallel Training Data using Static and Contextualized Embeddings](https://mlsanity.com/paper/2004.08728/) (2020)
- [A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languages](https://mlsanity.com/paper/2006.06202/) (2020)
- [The Tatoeba Translation Challenge -- Realistic Data Sets for Low Resource and Multilingual MT](https://mlsanity.com/paper/2010.06354/) (2020)
- [Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasing](https://mlsanity.com/paper/2004.14564/) (2020)
- [Learning Joint Multilingual Sentence Representations with Neural Machine Translation](https://mlsanity.com/paper/1704.04154/) (2017)
- [Cross-Lingual Semantic Role Labeling with High-Quality Translated Training Corpus](https://mlsanity.com/paper/2004.06295/) (2020)
- [Adversarial Deep Averaging Networks for Cross-Lingual Sentiment Classification](https://mlsanity.com/paper/1606.01614/) (2016)
- [Contextual Parameter Generation for Universal Neural Machine Translation](https://mlsanity.com/paper/1808.08493/) (2018)
- [1.5 billion words Arabic Corpus](https://mlsanity.com/paper/1611.04033/) (2016)

## Built on by (most cited first, 16 of 16)

- [Frustratingly Easy Label Projection for Cross-lingual Transfer](https://mlsanity.com/paper/2211.15613/) (2022)
- [DuReader_retrieval: A Large-scale Chinese Benchmark for Passage Retrieval from Web Search Engine](https://mlsanity.com/paper/2203.10232/) (2022)
- [Iterative Document-level Information Extraction via Imitation Learning](https://mlsanity.com/paper/2210.06600/) (2022)
- [MegaWika: Millions of reports and their sources across 50 diverse languages](https://mlsanity.com/paper/2307.07049/) (2023)
- [Contextual Label Projection for Cross-Lingual Structured Prediction](https://mlsanity.com/paper/2309.08943/) (2023)
- [Lost in Translation, Found in Spans: Identifying Claims in Multilingual Social Media](https://mlsanity.com/paper/2310.18205/) (2023)
- [Constrained Decoding for Cross-lingual Label Projection](https://mlsanity.com/paper/2402.03131/) (2024)
- [Por Qu'e N~ao Utiliser Alla Spraak? Mixed Training with Gradient Optimization in Few-Shot Cross-Lingual Transfer](https://mlsanity.com/paper/2204.13869/) (2022)
- [MultiTACRED: A Multilingual Version of the TAC Relation Extraction Dataset](https://mlsanity.com/paper/2305.04582/) (2023)
- [Multilingual Clinical NER: Translation or Cross-lingual Transfer?](https://mlsanity.com/paper/2306.04384/) (2023)
- [Zero-shot Cross-lingual Transfer is Under-specified Optimization](https://mlsanity.com/paper/2207.05666/) (2022)
- [MultiMUC: Multilingual Template Filling on MUC-4](https://mlsanity.com/paper/2401.16209/) (2024)
- [Multilingual Coreference Resolution in Multiparty Dialogue](https://mlsanity.com/paper/2208.01307/) (2022)
- [Language-Aware Multilingual Machine Translation with Self-Supervised Learning](https://mlsanity.com/paper/2302.05008/) (2023)
- [SHINE: Syntax-augmented Hierarchical Interactive Encoder for Zero-shot Cross-lingual Information Extraction](https://mlsanity.com/paper/2305.12389/) (2023)
- [Factuality Detection using Machine Translation -- a Use Case for German Clinical Text](https://mlsanity.com/paper/2308.08827/) (2023)

## Abstract (arXiv)

Zero-shot cross-lingual information extraction (IE) describes the construction of an IE model for some target language, given existing annotations exclusively in some other language, typically English. While the advance of pretrained multilingual encoders suggests an easy optimism of "train on English, run on any language", we find through a thorough exploration and extension of techniques that a combination of approaches, both new and old, leads to better performance than any one cross-lingual strategy in particular. We explore techniques including data projection and self-training, and how different pretrained encoders impact them. We use English-to-Arabic IE as our initial example, demonstrating strong performance in this setting for event extraction, named entity recognition, part-of-speech tagging, and dependency parsing. We then apply data projection and self-training to three tasks across eight target languages. Because no single set of techniques performs the best across all tasks, we encourage practitioners to explore various configurations of the techniques described in this work when seeking to improve on zero-shot training.