# Recent Advances in Neural Text Generation: A Task-Agnostic Survey

arXiv: https://arxiv.org/abs/2203.03047 · HTML: https://mlsanity.com/paper/2203.03047/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 174 of its 270 references that are in the map.
- First posted on arXiv 6 Mar 2022.

## Builds on (most cited first, 30 of 174)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [ConceptNet 5.5: An Open Multilingual Graph of General Knowledge](https://mlsanity.com/paper/1612.03975/) (2016)
- [DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation](https://mlsanity.com/paper/1911.00536/) (2019)
- [A Primer in BERTology: What we know about how BERT works](https://mlsanity.com/paper/2002.12327/) (2020)
- [Self-Attention Generative Adversarial Networks](https://mlsanity.com/paper/1805.08318/) (2018)
- [Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition](https://mlsanity.com/paper/1804.03209/) (2018)
- [Object Hallucination in Image Captioning](https://mlsanity.com/paper/1809.02156/) (2018)
- [Generating Sentences from a Continuous Space](https://mlsanity.com/paper/1511.06349/) (2015)
- [Pre-trained Models for Natural Language Processing: A Survey](https://mlsanity.com/paper/2003.08271/) (2020)
- [Neural Text Generation with Unlikelihood Training](https://mlsanity.com/paper/1908.04319/) (2019)
- [Texygen: A Benchmarking Platform for Text Generation Models](https://mlsanity.com/paper/1802.01886/) (2018)
- [MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance](https://mlsanity.com/paper/1909.02622/) (2019)
- [A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Stories](https://mlsanity.com/paper/1604.01696/) (2016)
- [SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient](https://mlsanity.com/paper/1609.05473/) (2016)
- [Wasserstein Auto-Encoders](https://mlsanity.com/paper/1711.01558/) (2017)
- [Unified Pre-training for Program Understanding and Generation](https://mlsanity.com/paper/2103.06333/) (2021)
- [Memory Networks](https://mlsanity.com/paper/1410.3916/) (2014)
- [MASS: Masked Sequence to Sequence Pre-training for Language Generation](https://mlsanity.com/paper/1905.02450/) (2019)

## Built on by (most cited first, 14 of 14)

- [FrameBERT: Conceptual Metaphor Detection with Frame Embedding Learning](https://mlsanity.com/paper/2302.04834/) (2023)
- [EtriCA: Event-Triggered Context-Aware Story Generation Augmented by Cross Attention](https://mlsanity.com/paper/2210.12463/) (2022)
- ["Why is this misleading?": Detecting News Headline Hallucinations with Explanations](https://mlsanity.com/paper/2302.05852/) (2023)
- [Enhancing Dialogue Generation via Dynamic Graph Knowledge Aggregation](https://mlsanity.com/paper/2306.16195/) (2023)
- [NGEP: A Graph-based Event Planning Framework for Story Generation](https://mlsanity.com/paper/2210.10602/) (2022)
- [Improving Chinese Story Generation via Awareness of Syntactic Dependencies and Semantics](https://mlsanity.com/paper/2210.10618/) (2022)
- [Improving Biomedical Abstractive Summarisation with Knowledge Aggregation from Citation Papers](https://mlsanity.com/paper/2310.15684/) (2023)
- [Terminology-aware Medical Dialogue Generation](https://mlsanity.com/paper/2210.15551/) (2022)
- [CTRLStruct: Dialogue Structure Learning for Open-Domain Response Generation](https://mlsanity.com/paper/2303.01094/) (2023)
- [CADGE: Context-Aware Dialogue Generation Enhanced with Graph-Structured Knowledge Aggregation](https://mlsanity.com/paper/2305.06294/) (2023)
- [A Cross-Attention Augmented Model for Event-Triggered Context-Aware Story Generation](https://mlsanity.com/paper/2311.11271/) (2023)
- [Beyond Generative Artificial Intelligence: Roadmap for Natural Language Generation](https://mlsanity.com/paper/2407.10554/) (2024)
- [Causal prompting model-based offline reinforcement learning](https://mlsanity.com/paper/2406.01065/) (2024)
- [ACL Anthology Helper: A Tool to Retrieve and Manage Literature from ACL Anthology](https://mlsanity.com/paper/2310.20467/) (2023)

## Abstract (arXiv)

In recent years, considerable research has been dedicated to the application of neural models in the field of natural language generation (NLG). The primary objective is to generate text that is both linguistically natural and human-like, while also exerting control over the generation process. This paper offers a comprehensive and task-agnostic survey of the recent advancements in neural text generation. These advancements have been facilitated through a multitude of developments, which we categorize into four key areas: data construction, neural frameworks, training and inference strategies, and evaluation metrics. By examining these different aspects, we aim to provide a holistic overview of the progress made in the field. Furthermore, we explore the future directions for the advancement of neural text generation, which encompass the utilization of neural pipelines and the incorporation of background knowledge. These avenues present promising opportunities to further enhance the capabilities of NLG systems. Overall, this survey serves to consolidate the current state of the art in neural text generation and highlights potential avenues for future research and development in this dynamic field.