mlsanityData to 7 Oct 2026

Multilingual Denoising Pre-training for Neural Machine Translation

arXiv 2001.08210 ↗First posted 22 Jan 2020cs.CLcited by 1,028

Not on the map. Why?

Abstract (arXiv)

This paper demonstrates that multilingual denoising pre-training produces significant performance gains across a wide variety of machine translation (MT) tasks. We present mBART -- a sequence-to-sequence denoising auto-encoder pre-trained on large-scale monolingual corpora in many languages using the BART objective. mBART is one of the first methods for pre-training a complete sequence-to-sequence model by denoising full texts in multiple languages, while previous approaches have focused only on the encoder, decoder, or reconstructing parts of the text. Pre-training a complete model allows it to be directly fine tuned for supervised (both sentence-level and document-level) and unsupervised machine translation, with no task-specific modifications. We demonstrate that adding mBART initialization produces performance gains in all but the highest-resource settings, including up to 12 BLEU points for low resource MT and over 5 BLEU points for many document-level and unsupervised models. We also show it also enables new types of transfer to language pairs with no bi-text or that were not in the pre-training corpus, and present extensive analysis of which factors contribute the most to effective pre-training.

Key facts. Cited by 1,028 papers in the map (data through 7 Oct 2026). First posted on arXiv 22 Jan 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 179; 2022: 250; 2023: 230; 2024: 193; 2025: 106; 2026: 70.

1,028 papers.

Sort:

By area:

  1. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  2. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  3. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  4. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  5. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  6. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 518
  7. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  8. 2023Scaling Speech Technology to 1,000+ Languagescited by 401
  9. 2021OCR-free Document Understanding Transformercited by 400
  10. 2021Unified Pre-training for Program Understanding and Generationcited by 361
  11. 2023How Good Are GPT Models at Machine Translation? A Comprehensive Evaluationcited by 344
  12. 2021Pre-Trained Models: Past, Present and Futurecited by 319
38 more
  1. 2021Few-shot Learning with Multilingual Language Modelscited by 278
  2. 2024Do Llamas Work in English? On the Latent Language of Multilingual Transformerscited by 275
  3. 2024MinerU: An Open-Source Solution for Precise Document Content Extractioncited by 255
  4. 2021The GEM Benchmark: Natural Language Generation, its Evaluation and Metricscited by 239
  5. 2023Language Model Tokenizers Introduce Unfairness Between Languagescited by 207
  6. 2023ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learningcited by 203
  7. 2023Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedbackcited by 189
  8. 2022Translation between Molecules and Natural Languagecited by 173
  9. 2023RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Modelscited by 173
  10. 2023Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Promptingcited by 163
  11. 2022Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuningcited by 149
  12. 2023SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialectscited by 149
  13. 2022MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languagescited by 142
  14. 2022mGPT: Few-Shot Learners Go Multilingualcited by 141
  15. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  16. 2022A Survey of Controllable Text Generation using Transformer-based Pre-trained Language Modelscited by 128
  17. 2022Two-Stream Network for Sign Language Recognition and Translationcited by 122
  18. 2022Pretrained Language Models for Text Generation: A Surveycited by 121
  19. 2022CLIP Models are Few-shot Learners: Empirical Studies on VQA and Visual Entailmentcited by 113
  20. 2021Contrastive Learning for Many-to-many Multilingual Neural Machine Translationcited by 108
  21. 2021Neural Machine Translation for Low-Resource Languages: A Surveycited by 108
  22. 2023SONAR: Sentence-Level Multimodal and Language-Agnostic Representationscited by 105
  23. 2023Gloss-free Sign Language Translation: Improving from Visual-Language Pretrainingcited by 104
  24. 2021LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understandingcited by 103
  25. 2023A Survey of Text Watermarking in the Era of Large Language Modelscited by 102
  26. 2023WebCPM: Interactive Web Search for Chinese Long-form Question Answeringcited by 93
  27. 2021Survey of Low-Resource Machine Translationcited by 92
  28. 2021Language Models are Few-shot Multilingual Learnerscited by 92
  29. 2024Fine-Tuned Language Models Generate Stable Inorganic Materials as Textcited by 92
  30. 2023Document-Level Machine Translation with Large Language Modelscited by 90
  31. 2021MultiEURLEX -- A multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfercited by 89
  32. 2023XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Modelscited by 88
  33. 2023Soft Merging of Experts with Adaptive Routingcited by 88
  34. 2023A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Modelscited by 88
  35. 2021Generate & Rank: A Multi-task Framework for Math Word Problemscited by 82
  36. 2023Modular Deep Learningcited by 82
  37. 2024A Survey on Multilingual Large Language Models: Corpora, Alignment, and Biascited by 82
  38. 2021CPT: A Pre-Trained Unbalanced Transformer for Both Chinese Language Understanding and Generationcited by 81

50 most cited of 1,028 All 1,028 (JSON)