# ERNIE-GEN: An Enhanced Multi-Flow Pre-training and Fine-tuning Framework for Natural Language Generation

arXiv: https://arxiv.org/abs/2001.11314 · HTML: https://mlsanity.com/paper/2001.11314/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Jan 2020.

## Built on by (most cited first, 30 of 40)

- [ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation](https://mlsanity.com/paper/2107.02137/) (2021)
- [Learning to Break the Loop: Analyzing and Mitigating Repetitions for Neural Text Generation](https://mlsanity.com/paper/2206.02369/) (2022)
- [ChatGPT Asks, BLIP-2 Answers: Automatic Questioning Towards Enriched Visual Descriptions](https://mlsanity.com/paper/2303.06594/) (2023)
- [BUFFET: Benchmarking Large Language Models for Few-shot Cross-lingual Transfer](https://mlsanity.com/paper/2305.14857/) (2023)
- [ERNIE-ViLG: Unified Generative Pre-training for Bidirectional Vision-Language Generation](https://mlsanity.com/paper/2112.15283/) (2021)
- [DialogVED: A Pre-trained Latent Variable Encoder-Decoder Model for Dialog Response Generation](https://mlsanity.com/paper/2204.13031/) (2022)
- [Large Language Models for UAVs: Current State and Pathways to the Future](https://mlsanity.com/paper/2405.01745/) (2024)
- [Generative Language Models for Paragraph-Level Question Generation](https://mlsanity.com/paper/2210.03992/) (2022)
- [GanLM: Encoder-Decoder Pre-training with an Auxiliary Discriminator](https://mlsanity.com/paper/2212.10218/) (2022)
- [Diaformer: Automatic Diagnosis via Symptoms Sequence Generation](https://mlsanity.com/paper/2112.10433/) (2021)
- [Selecting Better Samples from Pre-trained LLMs: A Case Study on Question Generation](https://mlsanity.com/paper/2209.11000/) (2022)
- [MVP: Multi-task Supervised Pre-training for Natural Language Generation](https://mlsanity.com/paper/2206.12131/) (2022)
- [Improve Query Focused Abstractive Summarization by Incorporating Answer Relevance](https://mlsanity.com/paper/2105.12969/) (2021)
- [Meta-Transfer Learning for Low-Resource Abstractive Summarization](https://mlsanity.com/paper/2102.09397/) (2021)
- [A Survey of Natural Language Generation](https://mlsanity.com/paper/2112.11739/) (2021)
- [Where to Go for the Holidays: Towards Mixed-Type Dialogs for Clarification of User Goals](https://mlsanity.com/paper/2204.07299/) (2022)
- [Scheduled Sampling Based on Decoding Steps for Neural Machine Translation](https://mlsanity.com/paper/2108.12963/) (2021)
- [Leaf: Multiple-Choice Question Generation](https://mlsanity.com/paper/2201.09012/) (2022)
- [On the Effectiveness of Pretrained Models for API Learning](https://mlsanity.com/paper/2204.03498/) (2022)
- [Enhance Incomplete Utterance Restoration by Joint Learning Token Extraction and Text Generation](https://mlsanity.com/paper/2204.03958/) (2022)
- [Transcormer: Transformer for Sentence Scoring with Sliding Language Modeling](https://mlsanity.com/paper/2205.12986/) (2022)
- [s2s-ft: Fine-Tuning Pretrained Transformer Encoders for Sequence-to-Sequence Learning](https://mlsanity.com/paper/2110.13640/) (2021)
- [Harnessing Abstractive Summarization for Fact-Checked Claim Detection](https://mlsanity.com/paper/2209.04612/) (2022)
- [A Survey on Neural Question Generation: Methods, Applications, and Prospects](https://mlsanity.com/paper/2402.18267/) (2024)
- [FRSUM: Towards Faithful Abstractive Summarization via Enhancing Factual Robustness](https://mlsanity.com/paper/2211.00294/) (2022)
- [A comprehensive review of automatic text summarization techniques: method, data, evaluation and coding](https://mlsanity.com/paper/2301.03403/) (2023)
- [RefleXGen:The unexamined code is not worth using](https://mlsanity.com/paper/2510.23674/) (2025)
- [Generating Answer Candidates for Quizzes and Answer-Aware Question Generators](https://mlsanity.com/paper/2108.12898/) (2021)
- [N-gram Is Back: Residual Learning of Neural Text Generation with n-gram Language Model](https://mlsanity.com/paper/2210.14431/) (2022)
- [Extract, Denoise and Enforce: Evaluating and Improving Concept Preservation for Text-to-Text Generation](https://mlsanity.com/paper/2104.08724/) (2021)

## Abstract (arXiv)

Current pre-training works in natural language generation pay little attention to the problem of exposure bias on downstream tasks. To address this issue, we propose an enhanced multi-flow sequence to sequence pre-training and fine-tuning framework named ERNIE-GEN, which bridges the discrepancy between training and inference with an infilling generation mechanism and a noise-aware generation method. To make generation closer to human writing patterns, this framework introduces a span-by-span generation flow that trains the model to predict semantically-complete spans consecutively rather than predicting word by word. Unlike existing pre-training methods, ERNIE-GEN incorporates multi-granularity target sampling to construct pre-training data, which enhances the correlation between encoder and decoder. Experimental results demonstrate that ERNIE-GEN achieves state-of-the-art results with a much smaller amount of pre-training data and parameters on a range of language generation tasks, including abstractive summarization (Gigaword and CNN/DailyMail), question generation (SQuAD), dialogue generation (Persona-Chat) and generative question answering (CoQA).