# Improving Zero-shot Multilingual Neural Machine Translation by Leveraging Cross-lingual Consistency Regularization

arXiv: https://arxiv.org/abs/2305.07310 · HTML: https://mlsanity.com/paper/2305.07310/

## Key facts

- Cited by 10 papers in the map (data through 7 Oct 2026).
- Builds on 27 of its 37 references that are in the map.
- First posted on arXiv 12 May 2023.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [Beyond English-Centric Multilingual Machine Translation](https://mlsanity.com/paper/2010.11125/) (2020)
- [Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation](https://mlsanity.com/paper/1611.04558/) (2016)
- [Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation](https://mlsanity.com/paper/2004.11867/) (2020)
- [Massively Multilingual Neural Machine Translation](https://mlsanity.com/paper/1903.00089/) (2019)
- [Multi-Way, Multilingual Neural Machine Translation with a Shared Attention Mechanism](https://mlsanity.com/paper/1601.01073/) (2016)
- [Contrastive Learning for Many-to-many Multilingual Neural Machine Translation](https://mlsanity.com/paper/2105.09501/) (2021)
- [Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder](https://mlsanity.com/paper/1611.04798/) (2016)
- [Universal Neural Machine Translation for Extremely Low Resource Languages](https://mlsanity.com/paper/1802.05368/) (2018)
- [Pre-training Multilingual Neural Machine Translation by Leveraging Alignment Information](https://mlsanity.com/paper/2010.03142/) (2020)
- [Improved Zero-shot Neural Machine Translation via Ignoring Spurious Correlations](https://mlsanity.com/paper/1906.01181/) (2019)
- [The Missing Ingredient in Zero-Shot Neural Machine Translation](https://mlsanity.com/paper/1903.07091/) (2019)
- [On Learning Universal Representations Across Languages](https://mlsanity.com/paper/2007.15960/) (2020)
- [Improving Zero-Shot Translation by Disentangling Positional Information](https://mlsanity.com/paper/2012.15127/) (2020)
- [Improving Zero-shot Translation with Language-Independent Constraints](https://mlsanity.com/paper/1906.08584/) (2019)
- [Improving Multilingual Translation by Representation and Gradient Regularization](https://mlsanity.com/paper/2109.04778/) (2021)
- [A neural interlingua for multilingual machine translation](https://mlsanity.com/paper/1804.08198/) (2018)
- [Language Tags Matter for Zero-Shot Neural Machine Translation](https://mlsanity.com/paper/2106.07930/) (2021)
- [Neural Machine Translation with Pivot Languages](https://mlsanity.com/paper/1611.04928/) (2016)
- [Zero-shot Cross-lingual Transfer of Neural Machine Translation with Multilingual Pretrained Encoders](https://mlsanity.com/paper/2104.08757/) (2021)
- [Cross-lingual Pre-training Based Transfer for Zero-shot Neural Machine Translation](https://mlsanity.com/paper/1912.01214/) (2019)
- [Consistency by Agreement in Zero-shot Neural Machine Translation](https://mlsanity.com/paper/1904.02338/) (2019)
- [Rethinking Zero-shot Neural Machine Translation: From a Perspective of Latent Variables](https://mlsanity.com/paper/2109.04705/) (2021)
- [Bi-SimCut: A Simple Strategy for Boosting Neural Machine Translation](https://mlsanity.com/paper/2206.02368/) (2022)
- [Triangular Architecture for Rare Language Translation](https://mlsanity.com/paper/1805.04813/) (2018)

## Built on by (most cited first, 10 of 10)

- [AdaMergeX: Cross-Lingual Transfer with Large Language Models via Adaptive Adapter Merging](https://mlsanity.com/paper/2402.18913/) (2024)
- [Pushing the Limits of Zero-shot End-to-End Speech Translation](https://mlsanity.com/paper/2402.10422/) (2024)
- [Transcending Language Boundaries: Harnessing LLMs for Low-Resource Language Translation](https://mlsanity.com/paper/2411.11295/) (2024)
- [Learning Multilingual Sentence Representations with Cross-lingual Consistency Regularization](https://mlsanity.com/paper/2306.06919/) (2023)
- [An Empirical Study of Consistency Regularization for End-to-End Speech-to-Text Translation](https://mlsanity.com/paper/2308.14482/) (2023)
- [How Multilingual Are Large Language Models Fine-Tuned for Translation?](https://mlsanity.com/paper/2405.20512/) (2024)
- [Towards Boosting Many-to-Many Multilingual Machine Translation with Large Language Models](https://mlsanity.com/paper/2401.05861/) (2024)
- [Languages Transferred Within the Encoder: On Representation Transfer in Zero-Shot Multilingual Translation](https://mlsanity.com/paper/2406.08092/) (2024)
- [LCS: A Language Converter Strategy for Zero-Shot Neural Machine Translation](https://mlsanity.com/paper/2406.02876/) (2024)
- [Vision-fused Attack: Advancing Aggressive and Stealthy Adversarial Text against Neural Machine Translation](https://mlsanity.com/paper/2409.05021/) (2024)

## Abstract (arXiv)

The multilingual neural machine translation (NMT) model has a promising capability of zero-shot translation, where it could directly translate between language pairs unseen during training. For good transfer performance from supervised directions to zero-shot directions, the multilingual NMT model is expected to learn universal representations across different languages. This paper introduces a cross-lingual consistency regularization, CrossConST, to bridge the representation gap among different languages and boost zero-shot translation performance. The theoretical analysis shows that CrossConST implicitly maximizes the probability distribution for zero-shot translation, and the experimental results on both low-resource and high-resource benchmarks show that CrossConST consistently improves the translation performance. The experimental analysis also proves that CrossConST could close the sentence representation gap and better align the representation space. Given the universality and simplicity of CrossConST, we believe it can serve as a strong baseline for future multilingual NMT research.