mlsanityData to 7 Oct 2026

ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training

arXiv 2001.04063 ↗First posted 13 Jan 2020cs.CLcited by 221

Not on the map. Why?

Abstract (arXiv)

This paper presents a new sequence-to-sequence pre-training model called ProphetNet, which introduces a novel self-supervised objective named future n-gram prediction and the proposed n-stream self-attention mechanism. Instead of optimizing one-step-ahead prediction in the traditional sequence-to-sequence model, the ProphetNet is optimized by n-step ahead prediction that predicts the next n tokens simultaneously based on previous context tokens at each time step. The future n-gram prediction explicitly encourages the model to plan for the future tokens and prevent overfitting on strong local correlations. We pre-train ProphetNet using a base scale dataset (16GB) and a large-scale dataset (160GB), respectively. Then we conduct experiments on CNN/DailyMail, Gigaword, and SQuAD 1.1 benchmarks for abstractive summarization and question generation tasks. Experimental results show that ProphetNet achieves new state-of-the-art results on all these datasets compared to the models using the same scale pre-training corpus.

Key facts. Cited by 221 papers in the map (data through 7 Oct 2026). First posted on arXiv 13 Jan 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 2; 2021: 55; 2022: 54; 2023: 41; 2024: 31; 2025: 20; 2026: 18.

221 papers.

Sort:

By area:

  1. 2020Learning to summarize from human feedbackcited by 2,562
  2. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  3. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 346
  4. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 344
  5. 2021R-Drop: Regularized Dropout for Neural Networkscited by 202
  6. 2022PromptCast: A New Prompt-based Learning Paradigm for Time Series Forecastingcited by 170
  7. 2024The pitfalls of next-token predictioncited by 156
  8. 2023mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Videocited by 154
  9. 2024Beyond Autoregression: Discrete Diffusion for Complex Reasoning and Planningcited by 135
  10. 2021SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarizationcited by 131
  11. 2022Pretrained Language Models for Text Generation: A Surveycited by 121
  12. 2023AR-Diffusion: Auto-Regressive Diffusion Model for Text Generationcited by 113
38 more
  1. 2022Learning to Break the Loop: Analyzing and Mitigating Repetitions for Neural Text Generationcited by 90
  2. 2023A Complete Survey on Generative AI (AIGC): Is ChatGPT from GPT-4 to GPT-5 All You Need?cited by 84
  3. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
  4. 2022Text Generation with Diffusion Language Models: A Pre-training Approach with Continuous Paragraph Denoisecited by 79
  5. 2021Medically Aware GPT-3 as a Data Generator for Medical Dialogue Summarizationcited by 76
  6. 2021Planning with Learned Entity Prompts for Abstractive Summarizationcited by 75
  7. 2022RankGen: Improving Text Generation with Large Ranking Modelscited by 72
  8. 2023A Comprehensive Survey on Applications of Transformers for Deep Learning Taskscited by 70
  9. 2020Text Generation by Learning from Demonstrationscited by 64
  10. 2021Poolingformer: Long Document Modeling with Pooling Attentioncited by 60
  11. 2025SuperBPE: Space Travel for Language Modelscited by 58
  12. 2023One Small Step for Generative AI, One Giant Leap for AGI: A Complete Survey on ChatGPT in AIGC Eracited by 55
  13. 2021A Survey on Green Deep Learningcited by 52
  14. 2024A Systematic Survey of Text Summarization: From Statistical Methods to Large Language Modelscited by 52
  15. 2021AdaptSum: Towards Low-Resource Domain Adaptation for Abstractive Summarizationcited by 51
  16. 2022NoisyTune: A Little Noise Can Help You Finetune Pretrained Language Models Bettercited by 43
  17. 2023CLIPSyntel: CLIP and LLM Synergy for Multimodal Question Summarization in Healthcarecited by 43
  18. 2025Over-Tokenized Transformer: Vocabulary is Generally Worth Scalingcited by 42
  19. 2022BanglaNLG and BanglaT5: Benchmarks and Resources for Evaluating Low-Resource Natural Language Generation in Banglacited by 40
  20. 2024Speculative Streaming: Fast LLM Inference without Auxiliary Modelscited by 37
  21. 2022DialogVED: A Pre-trained Latent Variable Encoder-Decoder Model for Dialog Response Generationcited by 35
  22. 2021BARTpho: Pre-trained Sequence-to-Sequence Models for Vietnamesecited by 33
  23. 2021Vision Guided Generative Pre-trained Language Models for Multimodal Abstractive Summarizationcited by 32
  24. 2021ProphetNet-X: Large-Scale Pre-training Models for English, Chinese, Multi-lingual, Dialog, and Code Generationcited by 31
  25. 2021Focus Attention: Promoting Faithfulness and Diversity in Summarizationcited by 30
  26. 2021Does Pretraining for Summarization Require Knowledge Transfer?cited by 30
  27. 2021MixQG: Neural Question Generation with Mixed Answer Typescited by 28
  28. 2022A Well-Composed Text is Half Done! Composition Sampling for Diverse Conditional Generationcited by 28
  29. 2022Generative Language Models for Paragraph-Level Question Generationcited by 28
  30. 2022NEWTS: A Corpus for News Topic-Focused Summarizationcited by 27
  31. 2022Wav2Seq: Pre-training Speech-to-Text Encoder-Decoder Models Using Pseudo Languagescited by 26
  32. 2024Stumbling Blocks: Stress Testing the Robustness of Machine-Generated Text Detectors Under Attackscited by 25
  33. 2023FACTIFY-5WQA: 5W Aspect-based Fact Verification through Question Answeringcited by 24
  34. 2021ISEEQ: Information Seeking Question Generation using Dynamic Meta-Information Retrieval and Knowledge Graphscited by 22
  35. 2021BASS: Boosting Abstractive Summarization with Unified Semantic Graphcited by 21
  36. 2021Reinforcement Learning for Abstractive Question Summarization with Question-aware Semantic Rewardscited by 21
  37. 2021Domain Adaptation with Pre-trained Transformers for Query Focused Abstractive Text Summarizationcited by 21
  38. 2023Diffusion-NAT: Self-Prompting Discrete Diffusion for Non-Autoregressive Text Generationcited by 21

50 most cited of 221 All 221 (JSON)