mlsanityData to 7 Oct 2026

Self-supervised and Supervised Joint Training for Resource-rich Machine Translation

arXiv 2106.04060 ↗First posted 8 Jun 2021cs.CLcited by 12

Not on the map. Why?

Abstract (arXiv)

Self-supervised pre-training of text representations has been successfully applied to low-resource Neural Machine Translation (NMT). However, it usually fails to achieve notable gains on resource-rich NMT. In this paper, we propose a joint training approach, F2-XEnDec, to combine self-supervised and supervised learning to optimize NMT models. To exploit complementary self-supervised signals for supervised learning, NMT models are trained on examples that are interbred from monolingual and parallel sentences through a new process called crossover encoder-decoder. Experiments on two resource-rich translation benchmarks, WMT'14 English-German and WMT'14 English-French, demonstrate that our approach achieves substantial improvements over several strong baseline methods and obtains a new state of the art of 46.19 BLEU on English-French when incorporating back translation. Results also show that our approach is capable of improving model robustness to input perturbations such as code-switching noise which frequently appears on social media.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 35 of its 48 references that are in the map. First posted on arXiv 8 Jun 2021.

Builds on

35 of 48 references.

References · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  4. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  5. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  6. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  7. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  8. 2018Deep contextualized word representationscited by 2,285
  9. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  10. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  11. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  12. 2019AugMix: A Simple Data Processing Method to Improve Robustness and Uncertaintycited by 897
23 more
  1. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 784
  2. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 779
  3. 2017Convolutional Sequence to Sequence Learningcited by 550
  4. 2019Confident Learning: Estimating Uncertainty in Dataset Labelscited by 411
  5. 2019MASS: Masked Sequence to Sequence Pre-training for Language Generationcited by 345
  6. 2019Unsupervised Label Noise Modeling and Loss Correctioncited by 327
  7. 2017Synthetic and Natural Noise Both Break Neural Machine Translationcited by 304
  8. 2017Unsupervised Machine Translation Using Monolingual Corpora Onlycited by 272
  9. 2018Scaling Neural Machine Translationcited by 204
  10. 2020MixText: Linguistically-Informed Interpolation of Hidden Space for Semi-Supervised Text Classificationcited by 186
  11. 2019Robust Neural Machine Translation with Doubly Adversarial Inputscited by 105
  12. 2019Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modelingcited by 86
  13. 2019Tagged Back-Translationcited by 76
  14. 2019Jointly Learning to Align and Translate with Transformer Modelscited by 75
  15. 2020Sequence-Level Mixed Sample Data Augmentationcited by 71
  16. 2020AdvAug: Robust Adversarial Augmentation for Neural Machine Translationcited by 66
  17. 2020Very Deep Transformers for Neural Machine Translationcited by 50
  18. 2016Unsupervised Pretraining for Sequence to Sequence Learningcited by 47
  19. 2019Towards Making the Most of BERT in Neural Machine Translationcited by 43
  20. 2021Faster Meta Update Strategy for Noise-Robust Deep Learningcited by 38
  21. 2016Semi-Supervised Learning for Neural Machine Translationcited by 36
  22. 2019Pre-trained Language Model Representations for Language Generationcited by 34
  23. 2019Data Diversification: A Simple Strategy For Neural Machine Translationcited by 6

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 6; 2023: 2; 2024: 2; 2025: 1.

12 papers.

  1. 2021Joint Unsupervised and Supervised Training for Multilingual ASRcited by 39
  2. 2022Spatial-Temporal Hypergraph Self-Supervised Learning for Crime Predictioncited by 39
  3. 2024Self-supervised Learning for Electroencephalogram: A Systematic Surveycited by 19
  4. 2023Cross-modality Data Augmentation for End-to-End Sign Language Translationcited by 16
  5. 2022Learning to Generalize to More: Continuous Semantic Augmentation for Neural Machine Translationcited by 13
  6. 2022BLISS: Robust Sequence-to-Sequence Learning via Self-Supervised Input Representationcited by 8
  7. 2022Multilingual Mix: Example Interpolation Improves Multilingual Neural Machine Translationcited by 7
  8. 2023Bilex Rx: Lexical Data Augmentation for Massively Multilingual Machine Translationcited by 7
  9. 2025ChA-MAEViT: Unifying Channel-Aware Masked Autoencoders and Multi-Channel Vision Transformers for Improved Cross-Channel Learningcited by 6
  10. 2022Improving Neural Machine Translation by Denoising Trainingcited by 4
  11. 2022Improving Zero-Shot Multilingual Translation with Universal Representations and Cross-Mappingscited by 4
  12. 2024Label-Free Model Failure Detection for Lidar-based Point Cloud Segmentationcited by 1