# Rethinking Zero-shot Neural Machine Translation: From a Perspective of Latent Variables

arXiv: https://arxiv.org/abs/2109.04705 · HTML: https://mlsanity.com/paper/2109.04705/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 25 references that are in the map.
- First posted on arXiv 10 Sep 2021.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019)
- [Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation](https://mlsanity.com/paper/1611.04558/) (2016)
- [MASS: Masked Sequence to Sequence Pre-training for Language Generation](https://mlsanity.com/paper/1905.02450/) (2019)
- [Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation](https://mlsanity.com/paper/2004.11867/) (2020)
- [Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges](https://mlsanity.com/paper/1907.05019/) (2019)
- [Multi-Way, Multilingual Neural Machine Translation with a Shared Attention Mechanism](https://mlsanity.com/paper/1601.01073/) (2016)
- [Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder](https://mlsanity.com/paper/1611.04798/) (2016)
- [Investigating Multilingual NMT Representations at Scale](https://mlsanity.com/paper/1909.02197/) (2019)
- [Improved Zero-shot Neural Machine Translation via Ignoring Spurious Correlations](https://mlsanity.com/paper/1906.01181/) (2019)
- [The Missing Ingredient in Zero-Shot Neural Machine Translation](https://mlsanity.com/paper/1903.07091/) (2019)
- [Improving Zero-Shot Translation by Disentangling Positional Information](https://mlsanity.com/paper/2012.15127/) (2020)
- [Cross-lingual Pre-training Based Transfer for Zero-shot Neural Machine Translation](https://mlsanity.com/paper/1912.01214/) (2019)
- [Triangular Architecture for Rare Language Translation](https://mlsanity.com/paper/1805.04813/) (2018)

## Built on by (most cited first, 14 of 14)

- [Improving Zero-shot Multilingual Neural Machine Translation by Leveraging Cross-lingual Consistency Regularization](https://mlsanity.com/paper/2305.07310/) (2023)
- [Towards a Better Understanding of Variations in Zero-Shot Neural Machine Translation Performance](https://mlsanity.com/paper/2310.10385/) (2023)
- [Understanding and Mitigating the Uncertainty in Zero-Shot Translation](https://mlsanity.com/paper/2205.10068/) (2022)
- [BranchNorm: Robustly Scaling Extremely Deep Transformers](https://mlsanity.com/paper/2305.02790/) (2023)
- [Improving Zero-Shot Multilingual Translation with Universal Representations and Cross-Mappings](https://mlsanity.com/paper/2210.15851/) (2022)
- [Exploring the Impact of Layer Normalization for Zero-shot Neural Machine Translation](https://mlsanity.com/paper/2305.09312/) (2023)
- [Unlikelihood Tuning on Negative Samples Amazingly Improves Zero-Shot Translation](https://mlsanity.com/paper/2309.16599/) (2023)
- [Improving Multilingual Neural Machine Translation by Utilizing Semantic and Linguistic Features](https://mlsanity.com/paper/2408.01394/) (2024)
- [Better Benchmarking LLMs for Zero-Shot Dependency Parsing](https://mlsanity.com/paper/2502.20866/) (2025)
- [TPDM: Selectively Removing Positional Information for Zero-shot Translation via Token-Level Position Disentangle Module](https://mlsanity.com/paper/2305.19857/) (2023)
- [LCS: A Language Converter Strategy for Zero-Shot Neural Machine Translation](https://mlsanity.com/paper/2406.02876/) (2024)
- [Variable-length Neural Interlingua Representations for Zero-shot Neural Machine Translation](https://mlsanity.com/paper/2305.10190/) (2023)
- [Direct Neural Machine Translation with Task-level Mixture of Experts models](https://mlsanity.com/paper/2310.12236/) (2023)
- [On the Shortcut Learning in Multilingual Neural Machine Translation](https://mlsanity.com/paper/2411.10581/) (2024)

## Abstract (arXiv)

Zero-shot translation, directly translating between language pairs unseen in training, is a promising capability of multilingual neural machine translation (NMT). However, it usually suffers from capturing spurious correlations between the output language and language invariant semantics due to the maximum likelihood training objective, leading to poor transfer performance on zero-shot translation. In this paper, we introduce a denoising autoencoder objective based on pivot language into traditional training objective to improve the translation accuracy on zero-shot directions. The theoretical analysis from the perspective of latent variables shows that our approach actually implicitly maximizes the probability distributions for zero-shot directions. On two benchmark machine translation datasets, we demonstrate that the proposed method is able to effectively eliminate the spurious correlations and significantly outperforms state-of-the-art methods with a remarkable performance. Our code is available at https://github.com/Victorwz/zs-nmt-dae.