mlsanityData to 7 Oct 2026

Zero-shot Cross-lingual Transfer of Neural Machine Translation with Multilingual Pretrained Encoders

arXiv 2104.08757 ↗First posted 18 Apr 2021cs.CLcited by 24

Not on the map. Why?

Abstract (arXiv)

Previous work mainly focuses on improving cross-lingual transfer for NLU tasks with a multilingual pretrained encoder (MPE), or improving the performance on supervised machine translation with BERT. However, it is under-explored that whether the MPE can help to facilitate the cross-lingual transferability of NMT model. In this paper, we focus on a zero-shot cross-lingual transfer task in NMT. In this task, the NMT model is trained with parallel dataset of only one language pair and an off-the-shelf MPE, then it is directly tested on zero-shot language pairs. We propose SixT, a simple yet effective model for this task. SixT leverages the MPE with a two-stage training schedule and gets further improvement with a position disentangled encoder and a capacity-enhanced decoder. Using this method, SixT significantly outperforms mBART, a pretrained multilingual encoder-decoder model explicitly designed for NMT, with an average improvement of 7.1 BLEU on zero-shot any-to-English test sets across 14 source languages. Furthermore, with much less training computation cost and training data, our model achieves better performance on 15 any-to-English test sets than CRISS and m2m-100, two strong multilingual NMT baselines.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 38 of its 46 references that are in the map. First posted on arXiv 18 Apr 2021.

Builds on

38 of 46 references.

References · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  5. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  6. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  7. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  8. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  9. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  10. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  11. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  12. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
26 more
  1. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  2. 2018Overcoming catastrophic forgetting with hard attention to the taskcited by 672
  3. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  4. 2020Beyond English-Centric Multilingual Machine Translationcited by 559
  5. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 486
  6. 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 467
  7. 2019MASS: Masked Sequence to Sequence Pre-training for Language Generationcited by 345
  8. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 330
  9. 2019Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERTcited by 301
  10. 2020Multilingual Translation with Extensible Multilingual Pretraining and Finetuningcited by 287
  11. 2020Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translationcited by 277
  12. 2019Massively Multilingual Neural Machine Translationcited by 246
  13. 2018Phrase-Based & Neural Unsupervised Machine Translationcited by 138
  14. 2018Meta-Learning for Low-Resource Neural Machine Translationcited by 105
  15. 2020Pre-training Multilingual Neural Machine Translation by Leveraging Alignment Informationcited by 64
  16. 2019Cross-Lingual Natural Language Generation via Pre-Trainingcited by 63
  17. 2020On Learning Universal Representations Across Languagescited by 58
  18. 2020Cross-lingual Retrieval for Iterative Self-Supervised Trainingcited by 54
  19. 2019Towards Making the Most of BERT in Neural Machine Translationcited by 43
  20. 2020Multi-SimLex: A Large-Scale Evaluation of Multilingual and Cross-Lingual Lexical Semantic Similaritycited by 38
  21. 2020Improving Zero-Shot Translation by Disentangling Positional Informationcited by 35
  22. 2019Acquiring Knowledge from Pre-trained Model to Neural Machine Translationcited by 30
  23. 2020Recipes for Adapting Pre-trained Monolingual and Multilingual Models to Machine Translationcited by 28
  24. 2020XLM-T: Scaling up Multilingual Machine Translation with Pretrained Cross-lingual Transformer Encoderscited by 24
  25. 2017A Teacher-Student Framework for Zero-Resource Neural Machine Translationcited by 22
  26. 2018Zero-Resource Neural Machine Translation with Multi-Agent Communication Gamecited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 6; 2023: 9; 2024: 4; 2025: 2; 2026: 1.

24 papers.

Sort:

By area:

  1. 2023Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Promptingcited by 163
  2. 2021Language Models are Few-shot Multilingual Learnerscited by 92
  3. 2025Beyond English: The Impact of Prompt Translation Strategies across Languages and Tasks in Multilingual LLMscited by 22
  4. 2023Vesper: A Compact and Effective Pretrained Model for Speech Emotion Recognitioncited by 16
  5. 2023CLIPTrans: Transferring Visual Knowledge with Pre-trained Models for Multimodal Machine Translationcited by 14
  6. 2023Improving Zero-shot Multilingual Neural Machine Translation by Leveraging Cross-lingual Consistency Regularizationcited by 10
  7. 2021Towards Making the Most of Multilingual Pretraining for Zero-Shot Neural Machine Translationcited by 8
  8. 2023μPLAN: Summarizing using a Content Plan as Cross-Lingual Bridgecited by 7
  9. 2023Neural Machine Translation for the Indigenous Languages of the Americas: An Introductioncited by 7
  10. 2024Can Machine Translation Bridge Multilingual Pretraining and Cross-lingual Transfer Learning?cited by 6
  11. 2022OCR Improves Machine Translation for Low-Resource Languagescited by 5
  12. 2024Adaptive Cross-lingual Text Classification through In-Context One-Shot Demonstrationscited by 5
12 more
  1. 2022Improving Zero-Shot Multilingual Translation with Universal Representations and Cross-Mappingscited by 4
  2. 2022Multilingual Bidirectional Unsupervised Translation Through Multilingual Finetuning and Back-Translationcited by 3
  3. 2023Self-Augmentation Improves Zero-Shot Cross-Lingual Transfercited by 3
  4. 2024Self-Translate-Train: Enhancing Cross-Lingual Transfer of Large Language Models via Inherent Capabilitycited by 3
  5. 2022Refining Low-Resource Unsupervised Translation by Language Disentanglement of Multilingual Modelcited by 2
  6. 2025Advances in LLMs with Focus on Reasoning, Adaptability, Efficiency and Ethicscited by 2
  7. 2022A Major Obstacle for NLP Research: Let's Talk about Time Allocation!cited by 1
  8. 2023Extrapolating Multilingual Understanding Models as Multilingual Generatorscited by 1
  9. 2024Enhancing Low-Resource NMT with a Multilingual Encoder and Knowledge Distillation: A Case Studycited by 1
  10. 2022Zero-shot Cross-lingual Conversational Semantic Role Labelingcited by 0
  11. 2023Train Global, Tailor Local: Minimalist Multilingual Translation into Endangered Languagescited by 0
  12. 2026Mix-MoE: Improving Multilingual Machine Translation of Large Language Models through Mixed MoEscited by 0