mlsanityData to 14 Oct 2026

Continual Mixed-Language Pre-Training for Extremely Low-Resource Neural Machine Translation

arXiv 2105.03953 ↗First posted 9 May 2021cs.CL cs.AIcited by 29

Not on the map. Why?

Abstract (arXiv)

The data scarcity in low-resource languages has become a bottleneck to building robust neural machine translation systems. Fine-tuning a multilingual pre-trained model (e.g., mBART (Liu et al., 2020)) on the translation task is a good approach for low-resource languages; however, its performance will be greatly limited when there are unseen languages in the translation pairs. In this paper, we present a continual pre-training (CPT) framework on mBART to effectively adapt it to unseen languages. We first construct noisy mixed-language text from the monolingual corpus of the target language in the translation pair to cover both the source and target languages, and then, we continue pre-training mBART to reconstruct the original monolingual text. Results show that our method can consistently improve the fine-tuning performance upon the mBART baseline, as well as other strong baselines, across all tested low-resource translation pairs containing unseen languages. Furthermore, our approach also boosts the performance on translation pairs where both languages are seen in the original mBART's pre-training. The code is available at https://github.com/zliucr/cpt-nmt.

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Builds on 47 of its 62 references that are in the map. First posted on arXiv 9 May 2021.

Builds on

47 of 62 references.

References · 47

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  7. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  8. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  9. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  10. 2019Cross-lingual Language Model Pretrainingcited by 1,090
  11. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,035
  12. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,026
35 more
  1. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 788
  2. 2019A Closer Look at Few-shot Classificationcited by 684
  3. 2020Beyond English-Centric Multilingual Machine Translationcited by 561
  4. 2017Word Translation Without Parallel Datacited by 487
  5. 2019MASS: Masked Sequence to Sequence Pre-training for Language Generationcited by 347
  6. 2020Multilingual Translation with Extensible Multilingual Pretraining and Finetuningcited by 287
  7. 2017Unsupervised Machine Translation Using Monolingual Corpora Onlycited by 273
  8. 2016Transfer Learning for Low-Resource Neural Machine Translationcited by 188
  9. 2017Unsupervised Neural Machine Translationcited by 149
  10. 2017Data Augmentation for Low-Resource Neural Machine Translationcited by 139
  11. 2018Phrase-Based & Neural Unsupervised Machine Translationcited by 138
  12. 2018Meta-Learning for Low-Resource Neural Machine Translationcited by 105
  13. 2020CrossNER: Evaluating Cross-Domain Named Entity Recognitioncited by 104
  14. 2019Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Taskscited by 96
  15. 2020Few-shot Natural Language Generation for Task-Oriented Dialogcited by 89
  16. 2020Extending Multilingual BERT to Low-Resource Languagescited by 88
  17. 2019Meta Relational Learning for Few-Shot Link Prediction in Knowledge Graphscited by 73
  18. 2018Universal Neural Machine Translation for Extremely Low Resource Languagescited by 66
  19. 2020Pre-training Multilingual Neural Machine Translation by Leveraging Alignment Informationcited by 64
  20. 2019The FLoRes Evaluation Datasets for Low-Resource Machine Translation: Nepali-English and Sinhala-Englishcited by 62
  21. 2021Are Multilingual Models Effective in Code-Switching?cited by 56
  22. 2019Generalized Data Augmentation for Low-Resource Translationcited by 53
  23. 2019Few-Shot NLG with Pre-Trained Language Modelcited by 52
  24. 2020Leveraging Monolingual Data with Self-Supervision for Multilingual Neural Machine Translationcited by 51
  25. 2021AdaptSum: Towards Low-Resource Domain Adaptation for Abstractive Summarizationcited by 51
  26. 2018Trivial Transfer Learning for Low-Resource Neural Machine Translationcited by 47
  27. 2020Code-switching pre-training for neural machine translationcited by 43
  28. 2019Attention-Informed Mixed-Language Training for Zero-shot Cross-lingual Task-oriented Dialogue Systemscited by 42
  29. 2020Language Model Prior for Low-Resource Neural Machine Translationcited by 37
  30. 2019Is Multilingual BERT Fluent in Language Generation?cited by 28
  31. 2020Learning Fast Adaptation on Cross-Accented Speech Recognitioncited by 28
  32. 2020Recipes for Adapting Pre-trained Monolingual and Multilingual Models to Machine Translationcited by 28
  33. 2019Exploiting Out-of-Domain Parallel Data through Multilingual Transfer Learning for Low-Resource Neural Machine Translationcited by 11
  34. 2019Incorporating Word and Subword Units in Unsupervised Machine Translation Using Language Model Rescoringcited by 7
  35. 2019Multilingual Neural Machine Translation for Zero-Resource Languagescited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 2; 2022: 6; 2023: 5; 2024: 5; 2025: 10; 2026: 1.

29 papers.

Sort:

By area:

  1. 2023Mitigating the Alignment Tax of RLHFcited by 183
  2. 2022Pretrained Language Models for Text Generation: A Surveycited by 122
  3. 2023Towards Geospatial Foundation Models via Continual Pretrainingcited by 109
  4. 2022A Few Thousand Translations Go a Long Way! Leveraging Pre-trained Models for African News Translationcited by 98
  5. 2021NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentationcited by 73
  6. 2024Multi-Label Guided Soft Contrastive Learning for Efficient Earth Observation Pretrainingcited by 28
  7. 2021AfroMT: Pretraining Strategies and Reproducible Benchmarks for Translation of 8 African Languagescited by 24
  8. 2022Pre-Trained Multilingual Sequence-to-Sequence Models: A Hope for Low-Resource Language Translation?cited by 24
  9. 2024ProxyLM: Predicting Language Model Performance on Multilingual Tasks via Proxy Modelscited by 20
  10. 2022DivEMT: Neural Machine Translation Post-Editing Effort Across Typologically Diverse Languagescited by 11
  11. 2023Bilex Rx: Lexical Data Augmentation for Massively Multilingual Machine Translationcited by 8
  12. 2025Lugha-Llama: Adapting Large Language Models for African Languagescited by 8
17 more
  1. 2025How Does Alignment Enhance LLMs' Multilingual Capabilities? A Language Neurons Perspectivecited by 8
  2. 2022Continual Learning of Neural Machine Translation within Low Forgetting Risk Regionscited by 6
  3. 2023Leveraging Auxiliary Domain Parallel Data in Intermediate Task Fine-tuning for Low-resource Translationcited by 5
  4. 2023A Comprehensive Evaluation of Parameter-Efficient Fine-Tuning on Software Engineering Taskscited by 5
  5. 2022Continual Knowledge Distillation for Neural Machine Translationcited by 4
  6. 2024Balancing Speciality and Versatility: A Coarse to Fine Framework for Mitigating Catastrophic Forgetting in Large Language Modelscited by 3
  7. 2024Cross-sensor self-supervised training and alignment for remote sensingcited by 3
  8. 2025Beyond Vanilla Fine-Tuning: Leveraging Multistage, Multilingual, and Domain-Specific Methods for Low-Resource Machine Translationcited by 2
  9. 2025Skin Lesion Phenotyping via Nested Multi-modal Contrastive Learningcited by 2
  10. 2024From Priest to Doctor: Domain Adaptation for Low-Resource Neural Machine Translationcited by 1
  11. 2025Adapting Multilingual Models to Code-Mixed Tasks via Model Mergingcited by 1
  12. 2025In-Domain African Languages Translation Using LLMs and Multi-armed Banditscited by 0
  13. 2025Adapting Language Models to Indonesian Local Languages: An Empirical Study of Language Transferability on Zero-Shot Settingscited by 0
  14. 2025HanjaBridge: Resolving Semantic Ambiguity in Korean LLMs via Hanja-Augmented Pre-Trainingcited by 0
  15. 2025VESSA: Video-based objEct-centric Self-Supervised Adaptation for Visual Foundation Modelscited by 0
  16. 2025Rethinking what Matters: Effective and Robust Multilingual Realignment for Low-Resource Languagescited by 0
  17. 2026Raising Bars, Not Parameters: LilMoo Compact Language Model for Hindicited by 0