mlsanityData to 7 Oct 2026

Understanding and Improving Sequence-to-Sequence Pretraining for Neural Machine Translation

arXiv 2203.08442 ↗First posted 16 Mar 2022cs.CL cs.AIcited by 14

Not on the map. Why?

Abstract (arXiv)

In this paper, we present a substantial step in better understanding the SOTA sequence-to-sequence (Seq2Seq) pretraining for neural machine translation~(NMT). We focus on studying the impact of the jointly pretrained decoder, which is the main difference between Seq2Seq pretraining and previous encoder-based pretraining approaches for NMT. By carefully designing experiments on three language pairs, we find that Seq2Seq pretraining is a double-edged sword: On one hand, it helps NMT models to produce more diverse translations and reduce adequacy-related translation errors. On the other hand, the discrepancies between Seq2Seq pretraining and NMT finetuning limit the translation quality (i.e., domain discrepancy) and induce the over-estimation issue (i.e., objective discrepancy). Based on these observations, we further propose simple and effective strategies, named in-domain pretraining and input adaptation to remedy the domain and objective discrepancies, respectively. Experimental results on several language pairs show that our approach can consistently improve both translation performance and model robustness upon Seq2Seq pretraining.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 27 of its 37 references that are in the map. First posted on arXiv 16 Mar 2022.

Builds on

27 of 37 references.

References · 27

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  5. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  6. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  7. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  8. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  9. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  10. 2019MASS: Masked Sequence to Sequence Pre-training for Language Generationcited by 345
  11. 2017Synthetic and Natural Noise Both Break Neural Machine Translationcited by 304
  12. 2020On Exposure Bias, Hallucination and Domain Shift in Neural Machine Translationcited by 127
15 more
  1. 2018Analyzing Uncertainty in Neural Machine Translationcited by 126
  2. 2021What to Pre-Train on? Efficient Intermediate Task Selectioncited by 62
  3. 2021Order-Agnostic Cross Entropy for Non-Autoregressive Machine Translationcited by 57
  4. 2020Understanding and Improving Lexical Choice in Non-Autoregressive Translationcited by 55
  5. 2018Denoising Neural Machine Translation Training with Trusted Data and Online Data Selectioncited by 43
  6. 2019Towards Making the Most of BERT in Neural Machine Translationcited by 43
  7. 2019Acquiring Knowledge from Pre-trained Model to Neural Machine Translationcited by 30
  8. 2016Neural Machine Translation with Reconstructioncited by 28
  9. 2020Data Rejuvenation: Exploiting Inactive Training Examples for Neural Machine Translationcited by 17
  10. 2021Self-Training Sampling with Monolingual Data Uncertainty for Neural Machine Translationcited by 17
  11. 2021On the Copying Behaviors of Pre-Training for Neural Machine Translationcited by 15
  12. 2021On the Influence of Masking Policies in Intermediate Pre-trainingcited by 12
  13. 2021On the Language Coverage Bias for Neural Machine Translationcited by 12
  14. 2016Context Gates for Neural Machine Translationcited by 10
  15. 2020Exploiting Unsupervised Data for Emotion Recognition in Conversationscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 9; 2024: 1; 2025: 1; 2026: 1.

14 papers.

Sort:

  1. 2023Exploring Human-Like Translation Strategy with Large Language Modelscited by 44
  2. 2023Transferable Adversarial Attacks on Vision Transformers with Token Gradient Regularizationcited by 42
  3. 2023Improving the Transferability of Adversarial Samples by Path-Augmented Methodcited by 36
  4. 2023MTTM: Metamorphic Testing for Textual Content Moderation Softwarecited by 20
  5. 2023CLIPTrans: Transferring Visual Knowledge with Pre-trained Models for Multimodal Machine Translationcited by 14
  6. 2023Bag of Tricks for Effective Language Model Pretraining and Downstream Adaptation: A Case Study on GLUEcited by 11
  7. 2023Disco-Bench: A Discourse-Aware Evaluation Benchmark for Language Modellingcited by 10
  8. 2022Vega-MT: The JD Explore Academy Translation System for WMT22cited by 9
  9. 2022Bridging Cross-Lingual Gaps During Leveraging the Multilingual Sequence-to-Sequence Pretraining for Text Generation and Understandingcited by 6
  10. 2023Leveraging Auxiliary Domain Parallel Data in Intermediate Task Fine-tuning for Low-resource Translationcited by 5
  11. 2023An Image is Worth a Thousand Toxic Words: A Metamorphic Testing Framework for Content Moderation Softwarecited by 3
  12. 2024On the Shortcut Learning in Multilingual Neural Machine Translationcited by 0
2 more
  1. 2025Metamorphic Testing for Audio Content Moderation Softwarecited by 0
  2. 2026Seeking Information with RAG-Assistants: Does Model Size Matter in Human-AI Collaborations?cited by 0