mlsanityData to 7 Oct 2026

Unsupervised Pretraining for Sequence to Sequence Learning

arXiv 1611.02683 ↗First posted 8 Nov 2016cs.CL cs.LG cs.NEcited by 47

Not on the map. Why?

Abstract (arXiv)

This work presents a general unsupervised learning method to improve the accuracy of sequence to sequence (seq2seq) models. In our method, the weights of the encoder and decoder of a seq2seq model are initialized with the pretrained weights of two language models and then fine-tuned with labeled data. We apply this method to challenging benchmarks in machine translation and abstractive summarization and find that it significantly improves the subsequent supervised models. Our main result is that pretraining improves the generalization of seq2seq models. We achieve state-of-the art results on the WMT EnglishrightarrowGerman task, surpassing a range of methods using both phrase-based machine translation and neural machine translation. Our method achieves a significant improvement of 1.3 BLEU from the previous best models on both WMT'14 and WMT'15 EnglishrightarrowGerman. We also conduct human evaluations on abstractive summarization and find that our method outperforms a purely supervised learning baseline in a statistically significant manner.

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). First posted on arXiv 8 Nov 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 20; 2022: 13; 2023: 6; 2024: 5; 2025: 2; 2026: 1.

47 papers.

Sort:

By area:

  1. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  2. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  3. 2021BERT: A Review of Applications in Natural Language Processing and Understandingcited by 109
  4. 2021Neural Machine Translation for Low-Resource Languages: A Surveycited by 108
  5. 2021Survey of Low-Resource Machine Translationcited by 92
  6. 2021Neural basis expansion analysis with exogenous variables: Forecasting electricity prices with NBEATSxcited by 63
  7. 2021The Effect of Domain and Diacritics in Yor`ub'a-English Neural Machine Translationcited by 38
  8. 2021INTERN: A New Learning Paradigm Towards General Visioncited by 33
  9. 2023A Review of Deep Learning for Video Captioningcited by 28
  10. 2022Uncertainty-DTW for Time Series and Sequencescited by 21
  11. 2022E2S2: Encoding-Enhanced Sequence-to-Sequence Pretraining for Language Understanding and Generationcited by 19
  12. 2024Self-Recognition in Language Modelscited by 17
35 more
  1. 2021The NLP Cookbook: Modern Recipes for Transformer based Deep Learning Architecturescited by 16
  2. 2021A Survey on Low-Resource Neural Machine Translationcited by 16
  3. 2024Finding the Missing Data: A BERT-inspired Approach Against Package Loss in Wireless Sensingcited by 15
  4. 2021Improving Hybrid CTC/Attention End-to-end Speech Recognition with Pretrained Acoustic and Language Modelcited by 14
  5. 2025Pre-Trained Policy Discriminators are General Reward Modelscited by 14
  6. 2021Self-supervised and Supervised Joint Training for Resource-rich Machine Translationcited by 12
  7. 2021Lexically Aware Semi-Supervised Learning for OCR Post-Correctioncited by 12
  8. 2022PARSE: Pairwise Alignment of Representations in Semi-Supervised EEG Learning for Emotion Recognitioncited by 12
  9. 2021BERTweetFR : Domain Adaptation of Pre-Trained Language Models for French Tweetscited by 9
  10. 2023Zero-Shot Learning for Requirements Classification: An Exploratory Studycited by 9
  11. 2024Soft Preference Optimization: Aligning Language Models to Expert Distributionscited by 9
  12. 2025Capturing Individual Human Preferences with Reward Featurescited by 8
  13. 2024Parameter Efficient Reinforcement Learning from Human Feedbackcited by 7
  14. 2021Code Generation from Natural Language with Less Prior and More Monolingual Datacited by 6
  15. 2021Cross-language Sentence Selection via Data Augmentation and Rationale Trainingcited by 5
  16. 2021Integrating Unsupervised Data Generation into Self-Supervised Neural Machine Translation for Low-Resource Languagescited by 4
  17. 2021Exploring Unsupervised Pretraining Objectives for Machine Translationcited by 3
  18. 2022Graph Enhanced BERT for Query Understandingcited by 3
  19. 2022A Survey on Sentence Embedding Models Performance for Patent Analysiscited by 3
  20. 2022Rethinking Clustering-Based Pseudo-Labeling for Unsupervised Meta-Learningcited by 3
  21. 2023Survey on Foundation Models for Prognostics and Health Management in Industrial Cyber-Physical Systemscited by 3
  22. 2021Neural Supervised Domain Adaptation by Augmenting Pre-trained Models with Random Unitscited by 2
  23. 2022Cost-Effective Training in Low-Resource Neural Machine Translationcited by 2
  24. 2023Improving Language Model Integration for Neural Machine Translationcited by 2
  25. 2022DBT-DMAE: An Effective Multivariate Time Series Pre-Train Model under Missing Datacited by 1
  26. 2026Ad Headline Generation using Self-Critical Masked Language Modelcited by 1
  27. 2021Joint Text and Label Generation for Spoken Language Understandingcited by 0
  28. 2021Noised Consistency Training for Text Summarizationcited by 0
  29. 2021One to Transfer All: A Universal Transfer Framework for Vision Foundation Model with Few Datacited by 0
  30. 2022DeepProphet2 -- A Deep Learning Gene Recommendation Enginecited by 0
  31. 2022Code Compliance Assessment as a Learning Problemcited by 0
  32. 2022Using Graph Algorithms to Pretrain Graph Completion Transformerscited by 0
  33. 2022Leveraging Pre-trained Models for Failure Analysis Triplets Generationcited by 0
  34. 2023Integrating Pre-trained Language Model into Neural Machine Translationcited by 0
  35. 2024SpacTor-T5: Pre-training T5 Models with Span Corruption and Replaced Token Detectioncited by 0