# Consistency by Agreement in Zero-shot Neural Machine Translation

arXiv: https://arxiv.org/abs/1904.02338 · HTML: https://mlsanity.com/paper/1904.02338/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Apr 2019.

## Built on by (most cited first, 22 of 22)

- [Contrastive Learning for Many-to-many Multilingual Neural Machine Translation](https://mlsanity.com/paper/2105.09501/) (2021)
- [State-of-the-art generalisation research in NLP: A taxonomy and review](https://mlsanity.com/paper/2210.03050/) (2022)
- [Eliciting the Translation Ability of Large Language Models via Multilingual Finetuning with Translation Instructions](https://mlsanity.com/paper/2305.15083/) (2023)
- [Language Tags Matter for Zero-Shot Neural Machine Translation](https://mlsanity.com/paper/2106.07930/) (2021)
- [Adapting High-resource NMT Models to Translate Low-resource Related Languages without Parallel Data](https://mlsanity.com/paper/2105.15071/) (2021)
- [A Survey on Low-Resource Neural Machine Translation](https://mlsanity.com/paper/2107.04239/) (2021)
- [Improving Zero-shot Multilingual Neural Machine Translation by Leveraging Cross-lingual Consistency Regularization](https://mlsanity.com/paper/2305.07310/) (2023)
- [Understanding and Mitigating the Uncertainty in Zero-Shot Translation](https://mlsanity.com/paper/2205.10068/) (2022)
- [Towards Making the Most of Multilingual Pretraining for Zero-Shot Neural Machine Translation](https://mlsanity.com/paper/2110.08547/) (2021)
- [Multilingual Mix: Example Interpolation Improves Multilingual Neural Machine Translation](https://mlsanity.com/paper/2203.07627/) (2022)
- ["Wikily" Supervised Neural Translation Tailored to Cross-Lingual Tasks](https://mlsanity.com/paper/2104.08384/) (2021)
- [Modeling Coverage for Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/2104.11897/) (2021)
- [Improving Zero-Shot Multilingual Translation with Universal Representations and Cross-Mappings](https://mlsanity.com/paper/2210.15851/) (2022)
- [Learning an Artificial Language for Knowledge-Sharing in Multilingual Translation](https://mlsanity.com/paper/2211.01292/) (2022)
- [Improving Zero-shot Neural Machine Translation on Language-specific Encoders-Decoders](https://mlsanity.com/paper/2102.06578/) (2021)
- [Exploring the Impact of Layer Normalization for Zero-shot Neural Machine Translation](https://mlsanity.com/paper/2305.09312/) (2023)
- [High-Dimensional Interlingual Representations of Large Language Models](https://mlsanity.com/paper/2503.11280/) (2025)
- [Self-Learning for Zero Shot Neural Machine Translation](https://mlsanity.com/paper/2103.05951/) (2021)
- [TPDM: Selectively Removing Positional Information for Zero-shot Translation via Token-Level Position Disentangle Module](https://mlsanity.com/paper/2305.19857/) (2023)
- [Variable-length Neural Interlingua Representations for Zero-shot Neural Machine Translation](https://mlsanity.com/paper/2305.10190/) (2023)
- [Direct Neural Machine Translation with Task-level Mixture of Experts models](https://mlsanity.com/paper/2310.12236/) (2023)
- [Defining Boundaries: The Impact of Domain Specification on Cross-Language and Cross-Domain Transfer in Machine Translation](https://mlsanity.com/paper/2408.11926/) (2024)

## Abstract (arXiv)

Generalization and reliability of multilingual translation often highly depend on the amount of available parallel data for each language pair of interest. In this paper, we focus on zero-shot generalization---a challenging setup that tests models on translation directions they have not been optimized for at training time. To solve the problem, we (i) reformulate multilingual translation as probabilistic inference, (ii) define the notion of zero-shot consistency and show why standard training often results in models unsuitable for zero-shot tasks, and (iii) introduce a consistent agreement-based training method that encourages the model to produce equivalent translations of parallel sentences in auxiliary languages. We test our multilingual NMT models on multiple public zero-shot translation benchmarks (IWSLT17, UN corpus, Europarl) and show that agreement-based learning often results in 2-3 BLEU zero-shot improvement over strong baselines without any loss in performance on supervised translation directions.