# Evaluating the Cross-Lingual Effectiveness of Massively Multilingual Neural Machine Translation

arXiv: https://arxiv.org/abs/1909.00437 · HTML: https://mlsanity.com/paper/1909.00437/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Sep 2019.

## Built on by (most cited first, 27 of 27)

- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [mSLAM: Massively multilingual joint pre-training for speech and text](https://mlsanity.com/paper/2202.01374/) (2022)
- [Enhancing Cross-lingual Transfer by Manifold Mixup](https://mlsanity.com/paper/2205.04182/) (2022)
- [Reuse Your Rewards: Reward Model Transfer for Zero-Shot Cross-Lingual Alignment](https://mlsanity.com/paper/2404.12318/) (2024)
- [Cross-Lingual Dialogue Dataset Creation via Outline-Based Generation](https://mlsanity.com/paper/2201.13405/) (2022)
- [nmT5 -- Is parallel data still relevant for pre-training massively multilingual language models?](https://mlsanity.com/paper/2106.02171/) (2021)
- [PARADISE: Exploiting Parallel Data for Multilingual Sequence-to-Sequence Pretraining](https://mlsanity.com/paper/2108.01887/) (2021)
- [Improving Zero-shot Cross-lingual Transfer between Closely Related Languages by injecting Character-level Noise](https://mlsanity.com/paper/2109.06772/) (2021)
- [Crossing the Conversational Chasm: A Primer on Natural Language Processing for Multilingual Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2104.08570/) (2021)
- [Translate & Fill: Improving Zero-Shot Multilingual Semantic Parsing with Synthetic Data](https://mlsanity.com/paper/2109.04319/) (2021)
- [Modelling Latent Translations for Cross-Lingual Transfer](https://mlsanity.com/paper/2107.11353/) (2021)
- [MergeDistill: Merging Pre-trained Language Models using Distillation](https://mlsanity.com/paper/2106.02834/) (2021)
- [Distilling Large Language Models into Tiny and Effective Students using pQRNN](https://mlsanity.com/paper/2101.08890/) (2021)
- [X-METRA-ADA: Cross-lingual Meta-Transfer Learning Adaptation to Natural Language Understanding and Question Answering](https://mlsanity.com/paper/2104.09696/) (2021)
- [Multilingual Document-Level Translation Enables Zero-Shot Transfer From Sentences to Documents](https://mlsanity.com/paper/2109.10341/) (2021)
- [CrossAligner & Co: Zero-Shot Transfer Methods for Task-Oriented Cross-lingual Natural Language Understanding](https://mlsanity.com/paper/2203.09982/) (2022)
- [Parameter Differentiation based Multilingual Neural Machine Translation](https://mlsanity.com/paper/2112.13619/) (2021)
- [Towards Unsupervised Content Disentanglement in Sentence Representations via Syntactic Roles](https://mlsanity.com/paper/2206.11184/) (2022)
- [Guardians of Discourse: Evaluating LLMs on Multilingual Offensive Language Detection](https://mlsanity.com/paper/2410.15623/) (2024)
- [On the Universality of Deep Contextual Language Models](https://mlsanity.com/paper/2109.07140/) (2021)
- [The Interpreter Understands Your Meaning: End-to-end Spoken Language Understanding Aided by Speech Translation](https://mlsanity.com/paper/2305.09652/) (2023)
- [Exploring Unsupervised Pretraining Objectives for Machine Translation](https://mlsanity.com/paper/2106.05634/) (2021)
- [Investigating Post-pretraining Representation Alignment for Cross-Lingual Question Answering](https://mlsanity.com/paper/2109.12028/) (2021)
- [Multi-level Distillation of Semantic Knowledge for Pre-training Multilingual Language Model](https://mlsanity.com/paper/2211.01200/) (2022)
- [Improving Cross-lingual Representation for Semantic Retrieval with Code-switching](https://mlsanity.com/paper/2403.01364/) (2024)
- [A Comparison of Language Modeling and Translation as Multilingual Pretraining Objectives](https://mlsanity.com/paper/2407.15489/) (2024)
- [Evaluating and explaining training strategies for zero-shot cross-lingual news sentiment analysis](https://mlsanity.com/paper/2409.20054/) (2024)

## Abstract (arXiv)

The recently proposed massively multilingual neural machine translation (NMT) system has been shown to be capable of translating over 100 languages to and from English within a single model. Its improved translation performance on low resource languages hints at potential cross-lingual transfer capability for downstream tasks. In this paper, we evaluate the cross-lingual effectiveness of representations from the encoder of a massively multilingual NMT model on 5 downstream classification and sequence labeling tasks covering a diverse set of over 50 languages. We compare against a strong baseline, multilingual BERT (mBERT), in different cross-lingual transfer learning scenarios and show gains in zero-shot transfer in 4 out of these 5 tasks.