mlsanityData to 7 Oct 2026

ProphetNet-X: Large-Scale Pre-training Models for English, Chinese, Multi-lingual, Dialog, and Code Generation

arXiv 2104.08006 ↗First posted 16 Apr 2021cs.CLcited by 31

Not on the map. Why?

Abstract (arXiv)

Now, the pre-training technique is ubiquitous in natural language processing field. ProphetNet is a pre-training based natural language generation method which shows powerful performance on English text summarization and question generation tasks. In this paper, we extend ProphetNet into other domains and languages, and present the ProphetNet family pre-training models, named ProphetNet-X, where X can be English, Chinese, Multi-lingual, and so on. We pre-train a cross-lingual generation model ProphetNet-Multi, a Chinese generation model ProphetNet-Zh, two open-domain dialog generation models ProphetNet-Dialog-En and ProphetNet-Dialog-Zh. And also, we provide a PLG (Programming Language Generation) model ProphetNet-Code to show the generation performance besides NLG (Natural Language Generation) tasks. In our experiments, ProphetNet-X models achieve new state-of-the-art performance on 10 benchmarks. All the models of ProphetNet-X share the same model structure, which allows users to easily switch between different models. We make the code and models publicly available, and we will keep updating more pre-training models and finetuning scripts.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Builds on 37 of its 46 references that are in the map. First posted on arXiv 16 Apr 2021.

Builds on

37 of 46 references.

References · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  8. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  9. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  10. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  11. 2015Teaching Machines to Read and Comprehendcited by 1,039
  12. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
25 more
  1. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,024
  2. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
  3. 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 820
  4. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  5. 2017DailyDialog: A Manually Labelled Multi-turn Dialogue Datasetcited by 649
  6. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 638
  7. 2020GraphCodeBERT: Pre-training Code Representations with Data Flowcited by 600
  8. 2019Text Summarization with Pretrained Encoderscited by 553
  9. 2015A Neural Attention Model for Abstractive Sentence Summarizationcited by 394
  10. 2021Unified Pre-training for Program Understanding and Generationcited by 361
  11. 2019MASS: Masked Sequence to Sequence Pre-training for Language Generationcited by 345
  12. 2020CLUE: A Chinese Language Understanding Evaluation Benchmarkcited by 277
  13. 2020ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-trainingcited by 221
  14. 2020XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generationcited by 221
  15. 2015A Neural Conversational Modelcited by 195
  16. 2019PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variablecited by 145
  17. 2015Neural Responding Machine for Short-Text Conversationcited by 124
  18. 2019Audio-Visual Scene-Aware Dialogcited by 122
  19. 2020A Large-Scale Chinese Short-Text Conversation Datasetcited by 77
  20. 2015LCSTS: A Large Scale Chinese Short Text Summarization Datasetcited by 65
  21. 2019Implicit Deep Latent Variable Models for Text Generationcited by 26
  22. 2018Global Encoding for Abstractive Summarizationcited by 16
  23. 2020Learning a Simple and Effective Model for Multi-turn Response Generation with Auxiliary Taskscited by 14
  24. 2020MATINF: A Jointly Labeled Large-Scale Dataset for Classification, Question Answering and Summarizationcited by 10
  25. 2018Query and Output: Generating Words by Querying Distributed Word Representations for Paraphrase Generationcited by 6

Built on by

Cited by year

Citing papers per year: 2021: 8; 2022: 14; 2023: 7; 2024: 0; 2025: 2.

31 papers.

Sort:

By area:

  1. 2023CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-Xcited by 294
  2. 2022SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarizationcited by 61
  3. 2022A Benchmark for Automatic Medical Consultation System: Frameworks, Tasks and Datasetscited by 45
  4. 2021Multilingual training for Software Engineeringcited by 43
  5. 2022Improving Sharpness-Aware Minimization with Fisher Mask for Better Generalization on Language Modelscited by 42
  6. 2022BanglaNLG and BanglaT5: Benchmarks and Resources for Evaluating Low-Resource Natural Language Generation in Banglacited by 40
  7. 2021PLATO-XL: Exploring the Large-scale Pre-training of Dialogue Generationcited by 37
  8. 2021BARTpho: Pre-trained Sequence-to-Sequence Models for Vietnamesecited by 33
  9. 2021Multimodal Dialogue Response Generationcited by 32
  10. 2022Faithful to the Document or to the World? Mitigating Hallucinations via Entity-linked Knowledge in Abstractive Summarizationcited by 20
  11. 2022Deep Learning Meets Software Engineering: A Survey on Pre-Trained Models of Source Codecited by 20
  12. 2022Recent Advances in Neural Text Generation: A Task-Agnostic Surveycited by 14
19 more
  1. 2021Code-switched inspired losses for generic spoken dialog representationscited by 10
  2. 2022On the Effectiveness of Pretrained Models for API Learningcited by 9
  3. 2023Response-act Guided Reinforced Dialogue Generation for Mental Health Counselingcited by 9
  4. 2021SYNFIX: Automatically Fixing Syntax Errors using Compiler Diagnosticscited by 8
  5. 2021Towards Making the Most of Multilingual Pretraining for Zero-Shot Neural Machine Translationcited by 8
  6. 2022Towards Summary Candidates Fusioncited by 8
  7. 2022Learning code summarization from a small and local datasetcited by 5
  8. 2022Joint Generator-Ranker Learning for Natural Language Generationcited by 4
  9. 2021Indian Legal NLP Benchmarks : A Surveycited by 3
  10. 2022A Survey on Artificial Intelligence for Source Code: A Dialogue Systems Perspectivecited by 3
  11. 2023HonestBait: Forward References for Attractive but Faithful Headline Generationcited by 3
  12. 2022PLATO-K: Internal and External Knowledge Enhanced Dialogue Generationcited by 2
  13. 2023Neuro-symbolic Zero-Shot Code Cloning with Cross-Language Intermediate Representationcited by 2
  14. 2022COOL, a Context Outlooker, and its Application to Question Answering and other Natural Language Processing Taskscited by 1
  15. 2023Knowledge Transfer for Pseudo-code Generation from Low Resource Programming Languagecited by 1
  16. 2023An Exploratory Literature Study on Sharing and Energy Use of Language Models for Source Codecited by 1
  17. 2023DocGen: Generating Detailed Parameter Docstrings in Pythoncited by 0
  18. 2025Interpreting Performance Profiles with Deep Learningcited by 0
  19. 2025NALA_MAINZ at BLP-2025 Task 2: A Multi-agent Approach for Bangla Instruction to Python Code Generationcited by 0