mlsanityData to 7 Oct 2026

MASS: Masked Sequence to Sequence Pre-training for Language Generation

arXiv 1905.02450 ↗First posted 7 May 2019cs.CL cs.AI cs.LGcited by 345

Not on the map. Why?

Abstract (arXiv)

Pre-training and fine-tuning, e.g., BERT, have achieved great success in language understanding by transferring knowledge from rich-resource pre-training task to the low/zero-resource downstream tasks. Inspired by the success of BERT, we propose MAsked Sequence to Sequence pre-training (MASS) for the encoder-decoder based language generation tasks. MASS adopts the encoder-decoder framework to reconstruct a sentence fragment given the remaining part of the sentence: its encoder takes a sentence with randomly masked fragment (several consecutive tokens) as input, and its decoder tries to predict this masked fragment. In this way, MASS can jointly train the encoder and decoder to develop the capability of representation extraction and language modeling. By further fine-tuning on a variety of zero/low-resource language generation tasks, including neural machine translation, text summarization and conversational response generation (3 tasks and totally 8 datasets), MASS achieves significant improvements over the baselines without pre-training or with other pre-training methods. Specially, we achieve the state-of-the-art accuracy (37.5 in terms of BLEU score) on the unsupervised English-French translation, even beating the early attention-based supervised model.

Key facts. Cited by 345 papers in the map (data through 7 Oct 2026). First posted on arXiv 7 May 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 143; 2022: 94; 2023: 55; 2024: 31; 2025: 16; 2026: 5.

345 papers.

Sort:

By area:

  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  2. 2020Learning to summarize from human feedbackcited by 2,562
  3. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  4. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  5. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  6. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
  7. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  8. 2024Large Language Models: A Surveycited by 506
  9. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  10. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  11. 2022Efficient Training of Language Models to Fill in the Middlecited by 248
  12. 2023A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPTcited by 221
38 more
  1. 2022HyperTree Proof Search for Neural Theorem Provingcited by 192
  2. 2023MADLAD-400: A Multilingual And Document-Level Large Audited Datasetcited by 180
  3. 2021The Values Encoded in Machine Learning Researchcited by 172
  4. 2023mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Videocited by 154
  5. 2021SCARF: Self-Supervised Contrastive Learning using Random Feature Corruptioncited by 151
  6. 2021OpenPrompt: An Open-source Framework for Prompt-learningcited by 148
  7. 2021A Unified Generative Framework for Various NER Subtaskscited by 141
  8. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  9. 2021Proof Artifact Co-training for Theorem Proving with Language Modelscited by 130
  10. 2021COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretrainingcited by 126
  11. 2022A Survey of Deep Learning for Mathematical Reasoningcited by 124
  12. 2022Pretrained Language Models for Text Generation: A Surveycited by 121
  13. 2022Should You Mask 15% in Masked Language Modeling?cited by 117
  14. 2021Contrastive Learning for Many-to-many Multilingual Neural Machine Translationcited by 108
  15. 2021A Simple Recipe for Multilingual Grammatical Error Correctioncited by 108
  16. 2021Neural Machine Translation for Low-Resource Languages: A Surveycited by 108
  17. 2021A Unified Generative Framework for Aspect-Based Sentiment Analysiscited by 97
  18. 2021Pretrained Language Models for Text Generation: A Surveycited by 95
  19. 2021Survey of Low-Resource Machine Translationcited by 92
  20. 2021M6: A Chinese Multimodal Pretrainercited by 91
  21. 2023The unreasonable effectiveness of few-shot learning for machine translationcited by 91
  22. 2021MusicBERT: Symbolic Music Understanding with Large-Scale Pre-Trainingcited by 89
  23. 2021E2E-VLP: End-to-End Vision-Language Pre-training Enhanced by Visual Learningcited by 84
  24. 2023A Complete Survey on Generative AI (AIGC): Is ChatGPT from GPT-4 to GPT-5 All You Need?cited by 84
  25. 2022Text Generation with Diffusion Language Models: A Pre-training Approach with Continuous Paragraph Denoisecited by 79
  26. 2021Planning with Learned Entity Prompts for Abstractive Summarizationcited by 75
  27. 2022Machine Generated Text: A Comprehensive Survey of Threat Models and Detection Methodscited by 75
  28. 2021XLM-E: Cross-lingual Language Model Pre-training via ELECTRAcited by 72
  29. 2021Spinning Language Models: Risks of Propaganda-As-A-Service and Countermeasurescited by 69
  30. 2022Building Machine Translation Systems for the Next Thousand Languagescited by 69
  31. 2021DOBF: A Deobfuscation Pre-Training Objective for Programming Languagescited by 68
  32. 2021ChineseBERT: Chinese Pretraining Enhanced by Glyph and Pinyin Informationcited by 65
  33. 2022AlexaTM 20B: Few-Shot Learning Using a Large-Scale Multilingual Seq2Seq Modelcited by 57
  34. 2023One Small Step for Generative AI, One Giant Leap for AGI: A Complete Survey on ChatGPT in AIGC Eracited by 55
  35. 2023A Survey on Time-Series Pre-Trained Modelscited by 52
  36. 2024A Systematic Survey of Text Summarization: From Statistical Methods to Large Language Modelscited by 52
  37. 2021DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoderscited by 50
  38. 2022What Do They Capture? -- A Structural Analysis of Pre-Trained Language Models for Source Codecited by 50

50 most cited of 345 All 345 (JSON)