# Improving Zero-shot Translation with Language-Independent Constraints

arXiv: https://arxiv.org/abs/1906.08584 · HTML: https://mlsanity.com/paper/1906.08584/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Jun 2019.

## Built on by (most cited first, 30 of 33)

- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Neural Machine Translation for Low-Resource Languages: A Survey](https://mlsanity.com/paper/2106.15115/) (2021)
- [SONAR: Sentence-Level Multimodal and Language-Agnostic Representations](https://mlsanity.com/paper/2308.11466/) (2023)
- [An Empirical Survey of Data Augmentation for Limited Data Learning in NLP](https://mlsanity.com/paper/2106.07499/) (2021)
- [Bi-Bimodal Modality Fusion for Correlation-Controlled Multimodal Sentiment Analysis](https://mlsanity.com/paper/2107.13669/) (2021)
- [Language Tags Matter for Zero-Shot Neural Machine Translation](https://mlsanity.com/paper/2106.07930/) (2021)
- [Middle-Layer Representation Alignment for Cross-Lingual Transfer in Fine-Tuned LLMs](https://mlsanity.com/paper/2502.14830/) (2025)
- [A Survey on Low-Resource Neural Machine Translation](https://mlsanity.com/paper/2107.04239/) (2021)
- [T-Modules: Translation Modules for Zero-Shot Cross-Modal Machine Translation](https://mlsanity.com/paper/2205.12216/) (2022)
- [Local Structure Matters Most: Perturbation Study in NLU](https://mlsanity.com/paper/2107.13955/) (2021)
- [Improving Zero-shot Multilingual Neural Machine Translation by Leveraging Cross-lingual Consistency Regularization](https://mlsanity.com/paper/2305.07310/) (2023)
- [Conditions for Catastrophic Forgetting in Multilingual Translation](https://mlsanity.com/paper/2510.19546/) (2025)
- [Towards Making the Most of Multilingual Pretraining for Zero-Shot Neural Machine Translation](https://mlsanity.com/paper/2110.08547/) (2021)
- [Adapting to Non-Centered Languages for Zero-shot Multilingual Translation](https://mlsanity.com/paper/2209.04138/) (2022)
- [Zero-shot Speech Translation](https://mlsanity.com/paper/2107.06010/) (2021)
- [Tackling data scarcity in speech translation using zero-shot multilingual machine translation techniques](https://mlsanity.com/paper/2201.11172/) (2022)
- [Improving Zero-Shot Multilingual Translation with Universal Representations and Cross-Mappings](https://mlsanity.com/paper/2210.15851/) (2022)
- [Learning an Artificial Language for Knowledge-Sharing in Multilingual Translation](https://mlsanity.com/paper/2211.01292/) (2022)
- [Exploring the Impact of Layer Normalization for Zero-shot Neural Machine Translation](https://mlsanity.com/paper/2305.09312/) (2023)
- [Gender Lost In Translation: How Bridging The Gap Between Languages Affects Gender Bias in Zero-Shot Multilingual Translation](https://mlsanity.com/paper/2305.16935/) (2023)
- [Languages Transferred Within the Encoder: On Representation Transfer in Zero-Shot Multilingual Translation](https://mlsanity.com/paper/2406.08092/) (2024)
- [Improving Multilingual Neural Machine Translation by Utilizing Semantic and Linguistic Features](https://mlsanity.com/paper/2408.01394/) (2024)
- [Self-Learning for Zero Shot Neural Machine Translation](https://mlsanity.com/paper/2103.05951/) (2021)
- [Unsupervised Transfer Learning in Multilingual Neural Machine Translation with Cross-Lingual Word Embeddings](https://mlsanity.com/paper/2103.06689/) (2021)
- [Informative Language Representation Learning for Massively Multilingual Neural Machine Translation](https://mlsanity.com/paper/2209.01530/) (2022)
- [Efficient Speech Translation with Pre-trained Models](https://mlsanity.com/paper/2211.04939/) (2022)
- [InterMulti:Multi-view Multimodal Interactions with Text-dominated Hierarchical High-order Fusion for Emotion Analysis](https://mlsanity.com/paper/2212.10030/) (2022)
- [Train Global, Tailor Local: Minimalist Multilingual Translation into Endangered Languages](https://mlsanity.com/paper/2305.03873/) (2023)
- [Variable-length Neural Interlingua Representations for Zero-shot Neural Machine Translation](https://mlsanity.com/paper/2305.10190/) (2023)
- [TCAN: Text-oriented Cross Attention Network for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2404.04545/) (2024)

## Abstract (arXiv)

An important concern in training multilingual neural machine translation (NMT) is to translate between language pairs unseen during training, i.e zero-shot translation. Improving this ability kills two birds with one stone by providing an alternative to pivot translation which also allows us to better understand how the model captures information between languages. In this work, we carried out an investigation on this capability of the multilingual NMT models. First, we intentionally create an encoder architecture which is independent with respect to the source language. Such experiments shed light on the ability of NMT encoders to learn multilingual representations, in general. Based on such proof of concept, we were able to design regularization methods into the standard Transformer model, so that the whole architecture becomes more robust in zero-shot conditions. We investigated the behaviour of such models on the standard IWSLT 2017 multilingual dataset. We achieved an average improvement of 2.23 BLEU points across 12 language pairs compared to the zero-shot performance of a state-of-the-art multilingual system. Additionally, we carry out further experiments in which the effect is confirmed even for language pairs with multiple intermediate pivots.