mlsanityData to 7 Oct 2026

On the Complementarity between Pre-Training and Random-Initialization for Resource-Rich Machine Translation

arXiv 2209.03316 ↗First posted 7 Sep 2022cs.CLcited by 15

Not on the map. Why?

Abstract (arXiv)

Pre-Training (PT) of text representations has been successfully applied to low-resource Neural Machine Translation (NMT). However, it usually fails to achieve notable gains (sometimes, even worse) on resource-rich NMT on par with its Random-Initialization (RI) counterpart. We take the first step to investigate the complementarity between PT and RI in resource-rich scenarios via two probing analyses, and find that: 1) PT improves NOT the accuracy, but the generalization by achieving flatter loss landscapes than that of RI; 2) PT improves NOT the confidence of lexical choice, but the negative diversity by assigning smoother lexical probability distributions than that of RI. Based on these insights, we propose to combine their complementarities with a model fusion algorithm that utilizes optimal transport to align neurons between PT and RI. Experiments on two resource-rich translation benchmarks, WMT'17 English-Chinese (20M) and WMT'19 English-German (36M), show that PT and RI could be nicely complementary to each other, achieving substantial improvements considering both translation accuracy, generalization, and negative diversity. Probing tools and code are released at: https://github.com/zanchangtong/PTvsRI.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 23 of its 29 references that are in the map. First posted on arXiv 7 Sep 2022.

Builds on

23 of 29 references.

References · 23

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  5. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  6. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  7. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  8. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  9. 2017Convolutional Sequence to Sequence Learningcited by 550
  10. 2019MASS: Masked Sequence to Sequence Pre-training for Language Generationcited by 345
  11. 2019Model Fusion via Optimal Transportcited by 217
  12. 2018Analyzing Uncertainty in Neural Machine Translationcited by 126
11 more
  1. 2019Visualizing and Understanding the Effectiveness of BERTcited by 67
  2. 2021Prevent the Language Model from being Overconfident in Neural Machine Translationcited by 34
  3. 2021Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translationcited by 34
  4. 2019Improving Neural Response Diversity with Frequency-Aware Cross-Entropy Losscited by 29
  5. 2019Modeling Recurrence for Transformercited by 26
  6. 2020Context-Aware Cross-Attention for Non-Autoregressive Translationcited by 26
  7. 2020Data Rejuvenation: Exploiting Inactive Training Examples for Neural Machine Translationcited by 17
  8. 2021Improving Neural Machine Translation by Bidirectional Trainingcited by 17
  9. 2021On the Copying Behaviors of Pre-Training for Neural Machine Translationcited by 15
  10. 2021On the Complementarity between Pre-Training and Back-Translation for Neural Machine Translationcited by 14
  11. 2022Learning to Generalize to More: Continuous Semantic Augmentation for Neural Machine Translationcited by 13

Built on by

Cited by year

Citing papers per year: 2022: 3; 2023: 8; 2024: 3; 2025: 1.

15 papers.

Sort:

By area:

  1. 2022SparseAdapter: An Easy Approach for Improving the Parameter-Efficiency of Adapterscited by 83
  2. 2025Towards LLM Unlearning Resilient to Relearning Attacks: A Sharpness-Aware Minimization Perspective and Beyondcited by 76
  3. 2022Improving Sharpness-Aware Minimization with Fisher Mask for Better Generalization on Language Modelscited by 42
  4. 2023Diversifying the Mixture-of-Experts Representation for Language Models with Orthogonal Optimizercited by 28
  5. 2023MerA: Merging Pretrained Adapters For Few-Shot Learningcited by 13
  6. 2023Bag of Tricks for Effective Language Model Pretraining and Downstream Adaptation: A Case Study on GLUEcited by 11
  7. 2022Vega-MT: The JD Explore Academy Translation System for WMT22cited by 9
  8. 2023Self-Evolution Learning for Discriminative Language Model Pretrainingcited by 8
  9. 2023Prompt-Learning for Cross-Lingual Relation Extractioncited by 4
  10. 2023Retrieval-based Knowledge Transfer: An Effective Approach for Extreme Large Language Model Compressioncited by 3
  11. 2023OmniForce: On Human-Centered, Large Model Empowered and Cloud-Edge Collaborative AutoML Systemcited by 2
  12. 2023Unlikelihood Tuning on Negative Samples Amazingly Improves Zero-Shot Translationcited by 2
3 more
  1. 2024The Impact of Syntactic and Semantic Proximity on Machine Translation with Back-Translationcited by 2
  2. 2024Why pre-training is beneficial for downstream classification tasks?cited by 1
  3. 2024Self-Evolution Knowledge Distillation for LLM-based Machine Translationcited by 1