# Language Tokens: A Frustratingly Simple Approach Improves Zero-Shot Performance of Multilingual Translation

arXiv: https://arxiv.org/abs/2208.05852 · HTML: https://mlsanity.com/paper/2208.05852/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 37 references that are in the map.
- First posted on arXiv 11 Aug 2022.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020)
- [Bag of Tricks for Efficient Text Classification](https://mlsanity.com/paper/1607.01759/) (2016)
- [On the Variance of the Adaptive Learning Rate and Beyond](https://mlsanity.com/paper/1908.03265/) (2019)
- [Beyond English-Centric Multilingual Machine Translation](https://mlsanity.com/paper/2010.11125/) (2020)
- [The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation](https://mlsanity.com/paper/2106.03193/) (2021)
- [Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation](https://mlsanity.com/paper/1611.04558/) (2016)
- [Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation](https://mlsanity.com/paper/2004.11867/) (2020)
- [Massively Multilingual Neural Machine Translation](https://mlsanity.com/paper/1903.00089/) (2019)
- [Simple, Scalable Adaptation for Neural Machine Translation](https://mlsanity.com/paper/1909.08478/) (2019)
- [CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEB](https://mlsanity.com/paper/1911.04944/) (2019)
- [CCAligned: A Massive Collection of Cross-Lingual Web-Document Pairs](https://mlsanity.com/paper/1911.06154/) (2019)
- [Achieving Human Parity on Automatic Chinese to English News Translation](https://mlsanity.com/paper/1803.05567/) (2018)
- [Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translation](https://mlsanity.com/paper/2006.10369/) (2020)
- [Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder](https://mlsanity.com/paper/1611.04798/) (2016)
- [Balancing Training for Multilingual Neural Machine Translation](https://mlsanity.com/paper/2004.06748/) (2020)
- [The Missing Ingredient in Zero-Shot Neural Machine Translation](https://mlsanity.com/paper/1903.07091/) (2019)
- [Facebook AI WMT21 News Translation Task Submission](https://mlsanity.com/paper/2108.03265/) (2021)
- [Learning Language Specific Sub-network for Multilingual Machine Translation](https://mlsanity.com/paper/2105.09259/) (2021)
- [Improving Multilingual Translation by Representation and Gradient Regularization](https://mlsanity.com/paper/2109.04778/) (2021)
- [Complete Multilingual Neural Machine Translation](https://mlsanity.com/paper/2010.10239/) (2020)
- [Subword Segmentation and a Single Bridge Language Affect Zero-Shot Neural Machine Translation](https://mlsanity.com/paper/2011.01703/) (2020)

## Built on by (most cited first, 1 of 1)

- [LAMASSU: Streaming Language-Agnostic Multilingual Speech Recognition and Translation Using Neural Transducers](https://mlsanity.com/paper/2211.02809/) (2022)

## Abstract (arXiv)

This paper proposes a simple yet effective method to improve direct (X-to-Y) translation for both cases: zero-shot and when direct data is available. We modify the input tokens at both the encoder and decoder to include signals for the source and target languages. We show a performance gain when training from scratch, or finetuning a pretrained model with the proposed setup. In the experiments, our method shows nearly 10.0 BLEU points gain on in-house datasets depending on the checkpoint selection criteria. In a WMT evaluation campaign, From-English performance improves by 4.17 and 2.87 BLEU points, in the zero-shot setting, and when direct data is available for training, respectively. While X-to-Y improves by 1.29 BLEU over the zero-shot baseline, and 0.44 over the many-to-many baseline. In the low-resource setting, we see a 1.5~1.7 point improvement when finetuning on X-to-Y domain data.