# Polyglot Contextual Representations Improve Crosslingual Transfer

arXiv: https://arxiv.org/abs/1902.09697 · HTML: https://mlsanity.com/paper/1902.09697/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Feb 2019.

## Built on by (most cited first, 16 of 16)

- [Language Models are Multilingual Chain-of-Thought Reasoners](https://mlsanity.com/paper/2210.03057/) (2022)
- [Evaluating GPT-4 and ChatGPT on Japanese Medical Licensing Examinations](https://mlsanity.com/paper/2303.18027/) (2023)
- [One Question Answering Model for Many Languages with Cross-lingual Dense Passage Retrieval](https://mlsanity.com/paper/2107.11976/) (2021)
- [MURAL: Multimodal, Multitask Retrieval Across Languages](https://mlsanity.com/paper/2109.05125/) (2021)
- [Reuse Your Rewards: Reward Model Transfer for Zero-Shot Cross-Lingual Alignment](https://mlsanity.com/paper/2404.12318/) (2024)
- [PARADISE: Exploiting Parallel Data for Multilingual Sequence-to-Sequence Pretraining](https://mlsanity.com/paper/2108.01887/) (2021)
- [Enhancing Multilingual Language Model with Massive Multilingual Knowledge Triples](https://mlsanity.com/paper/2111.10962/) (2021)
- [Soft Layer Selection with Meta-Learning for Zero-Shot Cross-Lingual Transfer](https://mlsanity.com/paper/2107.09840/) (2021)
- [Substructure Distribution Projection for Zero-Shot Cross-Lingual Dependency Parsing](https://mlsanity.com/paper/2110.08538/) (2021)
- [On the Relation between Syntactic Divergence and Zero-Shot Performance](https://mlsanity.com/paper/2110.04644/) (2021)
- [Cross-lingual Transferring of Pre-trained Contextualized Language Models](https://mlsanity.com/paper/2107.12627/) (2021)
- [Self-Translate-Train: Enhancing Cross-Lingual Transfer of Large Language Models via Inherent Capability](https://mlsanity.com/paper/2407.00454/) (2024)
- [A Knowledge-Enhanced Adversarial Model for Cross-lingual Structured Sentiment Analysis](https://mlsanity.com/paper/2205.15514/) (2022)
- [A survey of neural-network-based methods utilising comparable data for finding translation equivalents](https://mlsanity.com/paper/2410.15144/) (2024)
- [Can MLLMs Generalize to Multi-Party dialog? Exploring Multilingual Response Generation in Complex Scenarios](https://mlsanity.com/paper/2501.11269/) (2025)
- [EDU-NER-2025: Named Entity Recognition in Urdu Educational Texts using XLM-RoBERTa with X (formerly Twitter)](https://mlsanity.com/paper/2504.18142/) (2025)

## Abstract (arXiv)

We introduce Rosita, a method to produce multilingual contextual word representations by training a single language model on text from multiple languages. Our method combines the advantages of contextual word representations with those of multilingual representation learning. We produce language models from dissimilar language pairs (English/Arabic and English/Chinese) and use them in dependency parsing, semantic role labeling, and named entity recognition, with comparisons to monolingual and non-contextual variants. Our results provide further evidence for the benefits of polyglot learning, in which representations are shared across multiple languages.