# The Missing Ingredient in Zero-Shot Neural Machine Translation

arXiv: https://arxiv.org/abs/1903.07091 · HTML: https://mlsanity.com/paper/1903.07091/

## Key facts

- Cited by 59 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Mar 2019.

## Built on by (most cited first, 30 of 59)

- [Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languages](https://mlsanity.com/paper/2104.05596/) (2021)
- [Self-Supervised Multimodal Learning: A Survey](https://mlsanity.com/paper/2304.01008/) (2023)
- [Eliciting the Translation Ability of Large Language Models via Multilingual Finetuning with Translation Instructions](https://mlsanity.com/paper/2305.15083/) (2023)
- [Mitigating Hallucinations and Off-target Machine Translation with Source-Contrastive and Language-Contrastive Decoding](https://mlsanity.com/paper/2309.07098/) (2023)
- [SAMU-XLSR: Semantically-Aligned Multimodal Utterance-level Cross-Lingual Speech Representation](https://mlsanity.com/paper/2205.08180/) (2022)
- [Language Tags Matter for Zero-Shot Neural Machine Translation](https://mlsanity.com/paper/2106.07930/) (2021)
- [Middle-Layer Representation Alignment for Cross-Lingual Transfer in Fine-Tuned LLMs](https://mlsanity.com/paper/2502.14830/) (2025)
- [Using natural language prompts for machine translation](https://mlsanity.com/paper/2202.11822/) (2022)
- [Examining Scaling and Transfer of Language Model Architectures for Machine Translation](https://mlsanity.com/paper/2202.00528/) (2022)
- [Zero-Shot Cross-lingual Semantic Parsing](https://mlsanity.com/paper/2104.07554/) (2021)
- [Adapting High-resource NMT Models to Translate Low-resource Related Languages without Parallel Data](https://mlsanity.com/paper/2105.15071/) (2021)
- [A Survey on Low-Resource Neural Machine Translation](https://mlsanity.com/paper/2107.04239/) (2021)
- [T-Modules: Translation Modules for Zero-Shot Cross-Modal Machine Translation](https://mlsanity.com/paper/2205.12216/) (2022)
- [Rethinking Zero-shot Neural Machine Translation: From a Perspective of Latent Variables](https://mlsanity.com/paper/2109.04705/) (2021)
- [MMTAfrica: Multilingual Machine Translation for African Languages](https://mlsanity.com/paper/2204.04306/) (2022)
- [Improving Zero-shot Multilingual Neural Machine Translation by Leveraging Cross-lingual Consistency Regularization](https://mlsanity.com/paper/2305.07310/) (2023)
- [Towards a Better Understanding of Variations in Zero-Shot Neural Machine Translation Performance](https://mlsanity.com/paper/2310.10385/) (2023)
- [Conditions for Catastrophic Forgetting in Multilingual Translation](https://mlsanity.com/paper/2510.19546/) (2025)
- [Towards Making the Most of Multilingual Pretraining for Zero-Shot Neural Machine Translation](https://mlsanity.com/paper/2110.08547/) (2021)
- [Multilingual Document-Level Translation Enables Zero-Shot Transfer From Sentences to Documents](https://mlsanity.com/paper/2109.10341/) (2021)
- [Multilingual Mix: Example Interpolation Improves Multilingual Neural Machine Translation](https://mlsanity.com/paper/2203.07627/) (2022)
- [CrossAligner & Co: Zero-Shot Transfer Methods for Task-Oriented Cross-lingual Natural Language Understanding](https://mlsanity.com/paper/2203.09982/) (2022)
- [Neural Machine Translation for the Indigenous Languages of the Americas: An Introduction](https://mlsanity.com/paper/2306.06804/) (2023)
- [Adapting to Non-Centered Languages for Zero-shot Multilingual Translation](https://mlsanity.com/paper/2209.04138/) (2022)
- [Improving Multilingual Neural Machine Translation System for Indic Languages](https://mlsanity.com/paper/2209.13279/) (2022)
- [How Multilingual Are Large Language Models Fine-Tuned for Translation?](https://mlsanity.com/paper/2405.20512/) (2024)
- [Improving Zero-Shot Multilingual Translation with Universal Representations and Cross-Mappings](https://mlsanity.com/paper/2210.15851/) (2022)
- [Learning an Artificial Language for Knowledge-Sharing in Multilingual Translation](https://mlsanity.com/paper/2211.01292/) (2022)
- [The Interpreter Understands Your Meaning: End-to-end Spoken Language Understanding Aided by Speech Translation](https://mlsanity.com/paper/2305.09652/) (2023)
- [Improving Zero-shot Neural Machine Translation on Language-specific Encoders-Decoders](https://mlsanity.com/paper/2102.06578/) (2021)

## Abstract (arXiv)

Multilingual Neural Machine Translation (NMT) models are capable of translating between multiple source and target languages. Despite various approaches to train such models, they have difficulty with zero-shot translation: translating between language pairs that were not together seen during training. In this paper we first diagnose why state-of-the-art multilingual NMT models that rely purely on parameter sharing, fail to generalize to unseen language pairs. We then propose auxiliary losses on the NMT encoder that impose representational invariance across languages. Our simple approach vastly improves zero-shot translation quality without regressing on supervised directions. For the first time, on WMT14 English-FrenchGerman, we achieve zero-shot performance that is on par with pivoting. We also demonstrate the easy scalability of our approach to multiple languages on the IWSLT 2017 shared task.