mlsanityData to 14 Oct 2026

On the Complementarity between Pre-Training and Back-Translation for Neural Machine Translation

arXiv 2110.01811 ↗First posted 5 Oct 2021cs.CL cs.LGcited by 14

Not on the map. Why?

Abstract (arXiv)

Pre-training (PT) and back-translation (BT) are two simple and powerful methods to utilize monolingual data for improving the model performance of neural machine translation (NMT). This paper takes the first step to investigate the complementarity between PT and BT. We introduce two probing tasks for PT and BT respectively and find that PT mainly contributes to the encoder module while BT brings more benefits to the decoder. Experimental results show that PT and BT are nicely complementary to each other, establishing state-of-the-art performances on the WMT16 English-Romanian and English-Russian benchmarks. Through extensive analyses on sentence originality and word frequency, we also demonstrate that combining Tagged BT with PT is more helpful to their complementarity, leading to better translation quality. Source code is freely available at https://github.com/SunbowLiu/PTvsBT.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 37 of its 46 references that are in the map. First posted on arXiv 5 Oct 2021.

Builds on

37 of 46 references.

References · 37

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  4. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  6. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  7. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  8. 2019Cross-lingual Language Model Pretrainingcited by 1,090
  9. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,035
  10. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 788
  11. 2017Convolutional Sequence to Sequence Learningcited by 552
  12. 2019MASS: Masked Sequence to Sequence Pre-training for Language Generationcited by 347
25 more
  1. 2020Multilingual Translation with Extensible Multilingual Pretraining and Finetuningcited by 287
  2. 2019Leveraging Pre-trained Checkpoints for Sequence Generation Taskscited by 182
  3. 2019Revisiting Self-Training for Neural Sequence Generationcited by 155
  4. 2019Tagged Back-Translationcited by 77
  5. 2016Edinburgh Neural Machine Translation Systems for WMT 16cited by 76
  6. 2020Pre-training Multilingual Neural Machine Translation by Leveraging Alignment Informationcited by 64
  7. 2020Norm-Based Curriculum Learning for Neural Machine Translationcited by 60
  8. 2020Cross-lingual Retrieval for Iterative Self-Supervised Trainingcited by 54
  9. 2019The Effect of Translationese in Machine Translation Test Setscited by 49
  10. 2019compare-mt: A Tool for Holistic Comparison of Language Generation Systemscited by 43
  11. 2019Towards Making the Most of BERT in Neural Machine Translationcited by 43
  12. 2020Code-switching pre-training for neural machine translationcited by 43
  13. 2019On The Evaluation of Machine Translation Systems Trained With Back-Translationcited by 40
  14. 2019Improving Back-Translation with Uncertainty-based Confidence Estimationcited by 34
  15. 2021Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translationcited by 34
  16. 2021Meta-Curriculum Learning for Domain Adaptation in Neural Machine Translationcited by 28
  17. 2020Understanding and Improving Encoder Layer Fusion in Sequence-to-Sequence Learningcited by 26
  18. 2021Progressive Multi-Granularity Training for Non-Autoregressive Translationcited by 25
  19. 2018Back-Translation Sampling by Targeting Difficult Words in Neural Machine Translationcited by 19
  20. 2021TranSmart: A Practical Interactive Machine Translation Systemcited by 17
  21. 2021Self-Training Sampling with Monolingual Data Uncertainty for Neural Machine Translationcited by 17
  22. 2021Improving Neural Machine Translation by Bidirectional Trainingcited by 17
  23. 2021On the Copying Behaviors of Pre-Training for Neural Machine Translationcited by 15
  24. 2021On the Language Coverage Bias for Neural Machine Translationcited by 12
  25. 2019Encoders Help You Disambiguate Word Senses in Neural Machine Translationcited by 6

Built on by

Cited by year

Citing papers per year: 2022: 6; 2023: 5; 2024: 1; 2025: 2.

14 papers.

Sort:

By area:

  1. 2023On the Robustness of Aspect-based Sentiment Analysis: Rethinking Model, Data, and Trainingcited by 20
  2. 2022On the Complementarity between Pre-Training and Random-Initialization for Resource-Rich Machine Translationcited by 15
  3. 2022A Contrastive Cross-Channel Data Augmentation Framework for Aspect-based Sentiment Analysiscited by 12
  4. 2023Bag of Tricks for Effective Language Model Pretraining and Downstream Adaptation: A Case Study on GLUEcited by 11
  5. 2022Vega-MT: The JD Explore Academy Translation System for WMT22cited by 9
  6. 2023Back Translation for Speech-to-text Translation Without Transcriptscited by 9
  7. 2022Bridging Cross-Lingual Gaps During Leveraging the Multilingual Sequence-to-Sequence Pretraining for Text Generation and Understandingcited by 6
  8. 2022Breaking the Representation Bottleneck of Chinese Characters: Neural Machine Translation with Stroke Sequence Modelingcited by 6
  9. 2022ConsistTL: Modeling Consistency in Transfer Learning for Low-Resource Neural Machine Translationcited by 4
  10. 2025Lean-ing on Quality: How High-Quality Data Beats Diverse Multilingual Data in AutoFormalizationcited by 4
  11. 2023Retrieval-based Knowledge Transfer: An Effective Approach for Extreme Large Language Model Compressioncited by 3
  12. 2023When Does Monolingual Data Help Multilingual Translation: The Role of Domain and Model Scalecited by 2
2 more
  1. 2024Self-Evolution Knowledge Distillation for LLM-based Machine Translationcited by 1
  2. 2025A Representation Level Analysis of NMT Model Robustness to Grammatical Errorscited by 1