# Variable-length Neural Interlingua Representations for Zero-shot Neural Machine Translation

arXiv: https://arxiv.org/abs/2305.10190 · HTML: https://mlsanity.com/paper/2305.10190/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 33 references that are in the map.
- First posted on arXiv 17 May 2023.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [Mixed Precision Training](https://mlsanity.com/paper/1710.03740/) (2017)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization](https://mlsanity.com/paper/2003.11080/) (2020)
- [Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation](https://mlsanity.com/paper/1611.04558/) (2016)
- [Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation](https://mlsanity.com/paper/2004.11867/) (2020)
- [Massively Multilingual Neural Machine Translation](https://mlsanity.com/paper/1903.00089/) (2019)
- [Multi-Way, Multilingual Neural Machine Translation with a Shared Attention Mechanism](https://mlsanity.com/paper/1601.01073/) (2016)
- [Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder](https://mlsanity.com/paper/1611.04798/) (2016)
- [Improved Zero-shot Neural Machine Translation via Ignoring Spurious Correlations](https://mlsanity.com/paper/1906.01181/) (2019)
- [The Missing Ingredient in Zero-Shot Neural Machine Translation](https://mlsanity.com/paper/1903.07091/) (2019)
- [Multilingual Neural Machine Translation with Language Clustering](https://mlsanity.com/paper/1908.09324/) (2019)
- [Multi-Source Neural Translation](https://mlsanity.com/paper/1601.00710/) (2016)
- [Improving Zero-Shot Translation by Disentangling Positional Information](https://mlsanity.com/paper/2012.15127/) (2020)
- [Improving Zero-shot Translation with Language-Independent Constraints](https://mlsanity.com/paper/1906.08584/) (2019)
- [Improving Multilingual Translation by Representation and Gradient Regularization](https://mlsanity.com/paper/2109.04778/) (2021)
- [A neural interlingua for multilingual machine translation](https://mlsanity.com/paper/1804.08198/) (2018)
- [Language Tags Matter for Zero-Shot Neural Machine Translation](https://mlsanity.com/paper/2106.07930/) (2021)
- [Consistency by Agreement in Zero-shot Neural Machine Translation](https://mlsanity.com/paper/1904.02338/) (2019)
- [Multilingual Neural Machine Translation With Soft Decoupled Encoding](https://mlsanity.com/paper/1902.03499/) (2019)
- [Multilingual NMT with a language-independent attention bridge](https://mlsanity.com/paper/1811.00498/) (2018)
- [Rethinking Zero-shot Neural Machine Translation: From a Perspective of Latent Variables](https://mlsanity.com/paper/2109.04705/) (2021)
- [Exploring the Impact of Layer Normalization for Zero-shot Neural Machine Translation](https://mlsanity.com/paper/2305.09312/) (2023)

## Abstract (arXiv)

The language-independency of encoded representations within multilingual neural machine translation (MNMT) models is crucial for their generalization ability on zero-shot translation. Neural interlingua representations have been shown as an effective method for achieving this. However, fixed-length neural interlingua representations introduced in previous work can limit its flexibility and representation ability. In this study, we introduce a novel method to enhance neural interlingua representations by making their length variable, thereby overcoming the constraint of fixed-length neural interlingua representations. Our empirical results on zero-shot translation on OPUS, IWSLT, and Europarl datasets demonstrate stable model convergence and superior zero-shot translation results compared to fixed-length neural interlingua representations. However, our analysis reveals the suboptimal efficacy of our approach in translating from certain source languages, wherein we pinpoint the defective model component in our proposed method.