# Improved Zero-shot Neural Machine Translation via Ignoring Spurious Correlations

arXiv: https://arxiv.org/abs/1906.01181 · HTML: https://mlsanity.com/paper/1906.01181/

## Key facts

- Cited by 61 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Jun 2019.

## Built on by (most cited first, 30 of 61)

- [Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Models](https://mlsanity.com/paper/2402.16438/) (2024)
- [Neural Machine Translation for Low-Resource Languages: A Survey](https://mlsanity.com/paper/2106.15115/) (2021)
- [Contrastive Learning for Many-to-many Multilingual Neural Machine Translation](https://mlsanity.com/paper/2105.09501/) (2021)
- [A Close Look into the Calibration of Pre-trained Language Models](https://mlsanity.com/paper/2211.00151/) (2022)
- [Eliciting the Translation Ability of Large Language Models via Multilingual Finetuning with Translation Instructions](https://mlsanity.com/paper/2305.15083/) (2023)
- [SAMU-XLSR: Semantically-Aligned Multimodal Utterance-level Cross-Lingual Speech Representation](https://mlsanity.com/paper/2205.08180/) (2022)
- [Improving Multilingual Translation by Representation and Gradient Regularization](https://mlsanity.com/paper/2109.04778/) (2021)
- [Language Tags Matter for Zero-Shot Neural Machine Translation](https://mlsanity.com/paper/2106.07930/) (2021)
- [ZmBART: An Unsupervised Cross-lingual Transfer Framework for Language Generation](https://mlsanity.com/paper/2106.01597/) (2021)
- [Why Does Zero-Shot Cross-Lingual Generation Fail? An Explanation and a Solution](https://mlsanity.com/paper/2305.17325/) (2023)
- [Adapting High-resource NMT Models to Translate Low-resource Related Languages without Parallel Data](https://mlsanity.com/paper/2105.15071/) (2021)
- [Language Models are Good Translators](https://mlsanity.com/paper/2106.13627/) (2021)
- [Rethinking Zero-shot Neural Machine Translation: From a Perspective of Latent Variables](https://mlsanity.com/paper/2109.04705/) (2021)
- [Finding Memo: Extractive Memorization in Constrained Sequence Generation Tasks](https://mlsanity.com/paper/2210.12929/) (2022)
- [MMTAfrica: Multilingual Machine Translation for African Languages](https://mlsanity.com/paper/2204.04306/) (2022)
- [Improving Zero-shot Multilingual Neural Machine Translation by Leveraging Cross-lingual Consistency Regularization](https://mlsanity.com/paper/2305.07310/) (2023)
- [Towards a Better Understanding of Variations in Zero-Shot Neural Machine Translation Performance](https://mlsanity.com/paper/2310.10385/) (2023)
- [Conditions for Catastrophic Forgetting in Multilingual Translation](https://mlsanity.com/paper/2510.19546/) (2025)
- [Understanding and Mitigating the Uncertainty in Zero-Shot Translation](https://mlsanity.com/paper/2205.10068/) (2022)
- [Towards Making the Most of Multilingual Pretraining for Zero-Shot Neural Machine Translation](https://mlsanity.com/paper/2110.08547/) (2021)
- [Multilingual Document-Level Translation Enables Zero-Shot Transfer From Sentences to Documents](https://mlsanity.com/paper/2109.10341/) (2021)
- [FAST: Improving Controllability for Text Generation with Feedback Aware Self-Training](https://mlsanity.com/paper/2210.03167/) (2022)
- [Neural Machine Translation for the Indigenous Languages of the Americas: An Introduction](https://mlsanity.com/paper/2306.06804/) (2023)
- [Simultaneous Multi-Pivot Neural Machine Translation](https://mlsanity.com/paper/2104.07410/) (2021)
- [Learning to ignore: rethinking attention in CNNs](https://mlsanity.com/paper/2111.05684/) (2021)
- [Adapting to Non-Centered Languages for Zero-shot Multilingual Translation](https://mlsanity.com/paper/2209.04138/) (2022)
- [BranchNorm: Robustly Scaling Extremely Deep Transformers](https://mlsanity.com/paper/2305.02790/) (2023)
- [Mitigating Data Imbalance and Representation Degeneration in Multilingual Machine Translation](https://mlsanity.com/paper/2305.12786/) (2023)
- [Building Multilingual Machine Translation Systems That Serve Arbitrary X-Y Translations](https://mlsanity.com/paper/2206.14982/) (2022)
- [Improving Zero-Shot Multilingual Translation with Universal Representations and Cross-Mappings](https://mlsanity.com/paper/2210.15851/) (2022)

## Abstract (arXiv)

Zero-shot translation, translating between language pairs on which a Neural Machine Translation (NMT) system has never been trained, is an emergent property when training the system in multilingual settings. However, naive training for zero-shot NMT easily fails, and is sensitive to hyper-parameter setting. The performance typically lags far behind the more conventional pivot-based approach which translates twice using a third language as a pivot. In this work, we address the degeneracy problem due to capturing spurious correlations by quantitatively analyzing the mutual information between language IDs of the source and decoded sentences. Inspired by this analysis, we propose to use two simple but effective approaches: (1) decoder pre-training; (2) back-translation. These methods show significant improvement (4~22 BLEU points) over the vanilla zero-shot translation on three challenging multilingual datasets, and achieve similar or better results than the pivot-based approach.