# Improving Zero-Shot Translation by Disentangling Positional Information

arXiv: https://arxiv.org/abs/2012.15127 · HTML: https://mlsanity.com/paper/2012.15127/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Dec 2020.

## Built on by (most cited first, 30 of 35)

- [Position Information in Transformers: An Overview](https://mlsanity.com/paper/2102.11090/) (2021)
- [Improving Multilingual Translation by Representation and Gradient Regularization](https://mlsanity.com/paper/2109.04778/) (2021)
- [Language Tags Matter for Zero-Shot Neural Machine Translation](https://mlsanity.com/paper/2106.07930/) (2021)
- [Zero-shot Cross-lingual Transfer of Neural Machine Translation with Multilingual Pretrained Encoders](https://mlsanity.com/paper/2104.08757/) (2021)
- [Rethinking Zero-shot Neural Machine Translation: From a Perspective of Latent Variables](https://mlsanity.com/paper/2109.04705/) (2021)
- [Improving Zero-shot Multilingual Neural Machine Translation by Leveraging Cross-lingual Consistency Regularization](https://mlsanity.com/paper/2305.07310/) (2023)
- [Understanding and Mitigating the Uncertainty in Zero-Shot Translation](https://mlsanity.com/paper/2205.10068/) (2022)
- [How do Multimodal Foundation Models Encode Text and Speech? An Analysis of Cross-Lingual and Cross-Modal Representations](https://mlsanity.com/paper/2411.17666/) (2024)
- [Towards Making the Most of Multilingual Pretraining for Zero-Shot Neural Machine Translation](https://mlsanity.com/paper/2110.08547/) (2021)
- [Adapting to Non-Centered Languages for Zero-shot Multilingual Translation](https://mlsanity.com/paper/2209.04138/) (2022)
- [Zero-shot Speech Translation](https://mlsanity.com/paper/2107.06010/) (2021)
- [The Impact of Positional Encodings on Multilingual Compression](https://mlsanity.com/paper/2109.05388/) (2021)
- [Tackling data scarcity in speech translation using zero-shot multilingual machine translation techniques](https://mlsanity.com/paper/2201.11172/) (2022)
- [Improving Zero-Shot Multilingual Translation with Universal Representations and Cross-Mappings](https://mlsanity.com/paper/2210.15851/) (2022)
- [Learning an Artificial Language for Knowledge-Sharing in Multilingual Translation](https://mlsanity.com/paper/2211.01292/) (2022)
- [Exploring the Impact of Layer Normalization for Zero-shot Neural Machine Translation](https://mlsanity.com/paper/2305.09312/) (2023)
- [Gender Lost In Translation: How Bridging The Gap Between Languages Affects Gender Bias in Zero-Shot Multilingual Translation](https://mlsanity.com/paper/2305.16935/) (2023)
- [MT4CrossOIE: Multi-stage Tuning for Cross-lingual Open Information Extraction](https://mlsanity.com/paper/2308.06552/) (2023)
- [Tuning LLMs with Contrastive Alignment Instructions for Machine Translation in Unseen, Low-resource Languages](https://mlsanity.com/paper/2401.05811/) (2024)
- [Languages Transferred Within the Encoder: On Representation Transfer in Zero-Shot Multilingual Translation](https://mlsanity.com/paper/2406.08092/) (2024)
- [Robust Domain Adaptation for Pre-trained Multilingual Neural Machine Translation Models](https://mlsanity.com/paper/2210.14979/) (2022)
- [Out-of-Distribution Generalization in Text Classification: Past, Present, and Future](https://mlsanity.com/paper/2305.14104/) (2023)
- [TPDM: Selectively Removing Positional Information for Zero-shot Translation via Token-Level Position Disentangle Module](https://mlsanity.com/paper/2305.19857/) (2023)
- [EBBS: An Ensemble with Bi-Level Beam Search for Zero-Shot Machine Translation](https://mlsanity.com/paper/2403.00144/) (2024)
- [LCS: A Language Converter Strategy for Zero-Shot Neural Machine Translation](https://mlsanity.com/paper/2406.02876/) (2024)
- [Multilingual Non-Autoregressive Machine Translation without Knowledge Distillation](https://mlsanity.com/paper/2502.04537/) (2025)
- [Relaxed syntax modeling in Transformers for future-proof license plate recognition](https://mlsanity.com/paper/2506.17051/) (2025)
- [1Cademy at Semeval-2022 Task 1: Investigating the Effectiveness of Multilingual, Multitask, and Language-Agnostic Tricks for the Reverse Dictionary Task](https://mlsanity.com/paper/2206.03702/) (2022)
- [Variable-length Neural Interlingua Representations for Zero-shot Neural Machine Translation](https://mlsanity.com/paper/2305.10190/) (2023)
- [Direct Neural Machine Translation with Task-level Mixture of Experts models](https://mlsanity.com/paper/2310.12236/) (2023)

## Abstract (arXiv)

Multilingual neural machine translation has shown the capability of directly translating between language pairs unseen in training, i.e. zero-shot translation. Despite being conceptually attractive, it often suffers from low output quality. The difficulty of generalizing to new translation directions suggests the model representations are highly specific to those language pairs seen in training. We demonstrate that a main factor causing the language-specific representations is the positional correspondence to input tokens. We show that this can be easily alleviated by removing residual connections in an encoder layer. With this modification, we gain up to 18.5 BLEU points on zero-shot translation while retaining quality on supervised directions. The improvements are particularly prominent between related languages, where our proposed model outperforms pivot-based translation. Moreover, our approach allows easy integration of new languages, which substantially expands translation coverage. By thorough inspections of the hidden layer outputs, we show that our approach indeed leads to more language-independent representations.