mlsanityData to 7 Oct 2026

Towards a Better Understanding of Variations in Zero-Shot Neural Machine Translation Performance

arXiv 2310.10385 ↗First posted 16 Oct 2023cs.CL cs.LGcited by 10

Not on the map. Why?

Abstract (arXiv)

Multilingual Neural Machine Translation (MNMT) facilitates knowledge sharing but often suffers from poor zero-shot (ZS) translation qualities. While prior work has explored the causes of overall low ZS performance, our work introduces a fresh perspective: the presence of high variations in ZS performance. This suggests that MNMT does not uniformly exhibit poor ZS capability; instead, certain translation directions yield reasonable results. Through systematic experimentation involving 1,560 language directions spanning 40 languages, we identify three key factors contributing to high variations in ZS NMT performance: 1) target side translation capability 2) vocabulary overlap 3) linguistic properties. Our findings highlight that the target side translation quality is the most influential factor, with vocabulary overlap consistently impacting ZS performance. Additionally, linguistic properties, such as language family and writing system, play a role, particularly with smaller models. Furthermore, we suggest that the off-target issue is a symptom of inadequate ZS performance, emphasizing that zero-shot translation challenges extend beyond addressing the off-target problem. We release the data and models serving as a benchmark to study zero-shot for future research at https://github.com/Smu-Tan/ZS-NMT-Variations

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 38 of its 53 references that are in the map. First posted on arXiv 16 Oct 2023.

Builds on

38 of 53 references.

References · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  4. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  5. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  6. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  7. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  8. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  9. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 784
  10. 2019How multilingual is Multilingual BERT?cited by 642
  11. 2020Beyond English-Centric Multilingual Machine Translationcited by 559
  12. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 556
26 more
  1. 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 467
  2. 2016FastText.zip: Compressing text classification modelscited by 331
  3. 2020Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translationcited by 277
  4. 2019Massively Multilingual Neural Machine Translation in the Wild: Findings and Challengescited by 253
  5. 2019Massively Multilingual Neural Machine Translationcited by 246
  6. 2021Contrastive Learning for Many-to-many Multilingual Neural Machine Translationcited by 108
  7. 2020Pre-training Multilingual Neural Machine Translation by Leveraging Alignment Informationcited by 64
  8. 2019Improved Zero-shot Neural Machine Translation via Ignoring Spurious Correlationscited by 61
  9. 2019The Missing Ingredient in Zero-Shot Neural Machine Translationcited by 59
  10. 2023Towards a Common Understanding of Contributing Factors for Cross-Lingual Transfer in Multilingual Language Models: A Reviewcited by 50
  11. 2020Language Model Prior for Low-Resource Neural Machine Translationcited by 37
  12. 2021Improving Multilingual Translation by Representation and Gradient Regularizationcited by 31
  13. 2021Language Tags Matter for Zero-Shot Neural Machine Translationcited by 29
  14. 2022Pre-Trained Multilingual Sequence-to-Sequence Models: A Hope for Low-Resource Language Translation?cited by 24
  15. 2019Cross-lingual Pre-training Based Transfer for Zero-shot Neural Machine Translationcited by 23
  16. 2019Zero-shot Dependency Parsing with Pre-trained Multilingual Sentence Representationscited by 17
  17. 2023Viewing Knowledge Transfer in Multilingual Machine Translation Through a Representational Lenscited by 15
  18. 2021Rethinking Zero-shot Neural Machine Translation: From a Perspective of Latent Variablescited by 14
  19. 2021Alternative Input Signals Ease Transfer in Multilingual Machine Translationcited by 14
  20. 2019Target Conditioned Sampling: Optimizing Data Selection for Multilingual Neural Machine Translationcited by 11
  21. 2020On Romanization for Model Transfer Between Scripts in Neural Machine Translationcited by 9
  22. 2023Beyond Shared Vocabulary: Increasing Representational Word Similarities across Languages for Multilingual Machine Translationcited by 9
  23. 2021Towards Making the Most of Multilingual Pretraining for Zero-Shot Neural Machine Translationcited by 8
  24. 2022Adapting to Non-Centered Languages for Zero-shot Multilingual Translationcited by 6
  25. 2022Improving Zero-Shot Multilingual Translation with Universal Representations and Cross-Mappingscited by 4
  26. 2023CharSpan: Utilizing Lexical Similarity to Enable Zero-Shot Machine Translation for Extremely Low-resource Languagescited by 1

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 6; 2025: 3.

10 papers.

  1. 2024Neuron Specialization: Leveraging intrinsic task modularity for multilingual machine translationcited by 19
  2. 2025Remedy: Learning Machine Translation Evaluation from Human Preferences with Reward Modelingcited by 9
  3. 2025Mechanistic Understanding and Mitigation of Language Confusion in English-Centric Large Language Modelscited by 8
  4. 2024Languages Transferred Within the Encoder: On Representation Transfer in Zero-Shot Multilingual Translationcited by 3
  5. 2023Anti-LM Decoding for Zero-shot In-context Machine Translationcited by 2
  6. 2024Investigating the translation capabilities of Large Language Models trained on parallel data onlycited by 2
  7. 2024CoVoSwitch: Machine Translation of Synthetic Code-Switched Text Based on Intonation Unitscited by 1
  8. 2024How Far Can 100 Samples Go? Unlocking Overall Zero-Shot Multilingual Translation via Tiny Multi-Parallel Datacited by 0
  9. 2024Improving Language Transfer Capability of Decoder-only Architecture in Multilingual Neural Machine Translationcited by 0
  10. 2025Registering Source Tokens to Target Language Spaces in Multilingual Neural Machine Translationcited by 0