mlsanityData to 7 Oct 2026

Pretrained Language Models for Text Generation: A Survey

arXiv 2105.10311 ↗First posted 21 May 2021cs.CL cs.AIcited by 95

Not on the map. Why?

Abstract (arXiv)

Text generation has become one of the most important yet challenging tasks in natural language processing (NLP). The resurgence of deep learning has greatly advanced this field by neural generation models, especially the paradigm of pretrained language models (PLMs). In this paper, we present an overview of the major advances achieved in the topic of PLMs for text generation. As the preliminaries, we present the general task definition and briefly describe the mainstream architectures of PLMs for text generation. As the core content, we discuss how to adapt existing PLMs to model different input data and satisfy special properties in the generated text. We further summarize several important fine-tuning strategies for text generation. Finally, we present several future directions and conclude this paper. Our survey aims to provide text generation researchers a synthesis and pointer to related research.

Key facts. Cited by 95 papers in the map (data through 7 Oct 2026). Builds on 48 of its 61 references that are in the map. First posted on arXiv 21 May 2021.

Builds on

48 of 61 references.

References · 48

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  8. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  9. 2018Deep contextualized word representationscited by 2,285
  10. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  11. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  12. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
36 more
  1. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 754
  2. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  3. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  4. 2019Text Summarization with Pretrained Encoderscited by 553
  5. 2017Convolutional Sequence to Sequence Learningcited by 550
  6. 2019ERNIE: Enhanced Language Representation with Informative Entitiescited by 525
  7. 2020Pre-trained Models for Natural Language Processing: A Surveycited by 504
  8. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  9. 2019MASS: Masked Sequence to Sequence Pre-training for Language Generationcited by 345
  10. 2020Pretrained Transformers Improve Out-of-Distribution Robustnesscited by 293
  11. 2019TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agentscited by 188
  12. 2019Leveraging Pre-trained Checkpoints for Sequence Generation Taskscited by 182
  13. 2019HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarizationcited by 120
  14. 2020Investigating Pretrained Language Models for Graph-to-Text Generationcited by 117
  15. 2020Knowledge-Grounded Dialogue Generation with Pre-trained Language Modelscited by 107
  16. 2019Discourse-Aware Neural Extractive Text Summarizationcited by 104
  17. 2020PLATO-2: Towards Building an Open-Domain Chatbot via Curriculum Learningcited by 79
  18. 2019Sentence Centrality Revisited for Unsupervised Summarizationcited by 64
  19. 2020Pre-training Multilingual Neural Machine Translation by Leveraging Alignment Informationcited by 64
  20. 2019Few-Shot NLG with Pre-Trained Language Modelcited by 52
  21. 2020XGPT: Cross-modal Generative Pre-Training for Image Captioningcited by 51
  22. 2020Code-switching pre-training for neural machine translationcited by 43
  23. 2019Pretraining-Based Natural Language Generation for Text Summarizationcited by 40
  24. 2019Learning Video Representations using Contrastive Bidirectional Transformercited by 39
  25. 2020GPT-too: A language-model-first approach for AMR-to-text generationcited by 38
  26. 2018Improving Abstraction in Text Summarizationcited by 35
  27. 2020TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoisingcited by 30
  28. 2021A Short Survey of Pre-trained Language Models for Conversational AI-A NewAge in NLPcited by 25
  29. 2020Unsupervised Extractive Summarization by Pre-training Hierarchical Transformerscited by 16
  30. 2021TextBox: A Unified, Modularized, and Extensible Framework for Text Generationcited by 15
  31. 2020Knowledge-Enhanced Personalized Review Generation with Capsule Graph Neural Networkcited by 14
  32. 2019Generating Long and Informative Reviews with Aspect-Aware Coarse-to-Fine Decodingcited by 13
  33. 2021Knowledge-based Review Generation by Coherence Enhanced Text Planningcited by 10
  34. 2019Unsupervised pre-training for sequence to sequence speech recognitioncited by 6
  35. 2018Unsupervised Cross-lingual Word Embedding by Multilingual Neural Language Modelscited by 4
  36. 2020A Simple and Efficient Multi-Task Learning Approach for Conditioned Dialogue Generationcited by 2

Built on by

Cited by year

Citing papers per year: 2021: 5; 2022: 21; 2023: 32; 2024: 27; 2025: 10.

95 papers.

Sort:

By area:

  1. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,107
  2. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  3. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 663
  4. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  5. 2023RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Modelscited by 272
  6. 2023A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPTcited by 221
  7. 2022Pretrained Language Models for Text Generation: A Surveycited by 121
  8. 2024AutoAttacker: A Large Language Model Guided System to Implement Automatic Cyber-attackscited by 94
  9. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  10. 2023A Survey of Safety and Trustworthiness of Large Language Models through the Lens of Verification and Validationcited by 91
  11. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 71
  12. 2024Raidar: geneRative AI Detection viA Rewritingcited by 51
38 more
  1. 2023Diffusion Models for Non-autoregressive Text Generation: A Surveycited by 43
  2. 2022Knowledge Graph Embedding: A Survey from the Perspective of Representation Spacescited by 41
  3. 2022Attribution and Obfuscation of Neural Text Authorship: A Data Mining Perspectivecited by 40
  4. 2024Enhancing Multilingual Capabilities of Large Language Models through Self-Distillation from Resource-Rich Languagescited by 36
  5. 2024Reward Guided Latent Consistency Distillationcited by 35
  6. 2023Multi-Task Instruction Tuning of LLaMa for Specific Scenarios: A Preliminary Study on Writing Assistancecited by 33
  7. 2022Metaphors in Pre-Trained Language Models: Probing and Generalization Across Datasets and Languagescited by 30
  8. 2023Task-Optimized Adapters for an End-to-End Task-Oriented Dialogue Systemcited by 30
  9. 2021Few-shot Knowledge Graph-to-Text Generation with Pretrained Language Modelscited by 29
  10. 2023ChatGPT Needs SPADE (Sustainability, PrivAcy, Digital divide, and Ethics) Evaluation: A Reviewcited by 28
  11. 2022Analogy Generation by Prompting Large Language Models: A Case Study of InstructGPTcited by 27
  12. 2022ByGPT5: End-to-End Style-conditioned Poetry Generation with Token-free Language Modelscited by 25
  13. 2022On the Domain Adaptation and Generalization of Pretrained Language Models: A Surveycited by 22
  14. 2023LLM for SoC Security: A Paradigm Shiftcited by 22
  15. 2023Diffusion-NAT: Self-Prompting Discrete Diffusion for Non-Autoregressive Text Generationcited by 21
  16. 2023Multimodality of AI for Education: Towards Artificial General Intelligencecited by 21
  17. 2023Bootstrapping Interactive Image-Text Alignment for Remote Sensing Image Captioningcited by 18
  18. 2023User Modeling in the Era of Large Language Models: Current Research and Future Directionscited by 18
  19. 2024Text Generation: A Systematic Literature Review of Tasks, Evaluation, and Challengescited by 18
  20. 2024Exploring the Frontiers of LLMs in Psychological Applications: A Comprehensive Reviewcited by 17
  21. 2023Self-supervised Meta-Prompt Learning with Meta-Gradient Regularization for Few-shot Generalizationcited by 15
  22. 2023Challenges and Remedies to Privacy and Security in AIGC: Exploring the Potential of Privacy Computing, Blockchain, and Beyondcited by 14
  23. 2024Collu-Bench: A Benchmark for Predicting Language Model Hallucinations in Codecited by 13
  24. 2023Mastering Symbolic Operations: Augmenting Language Models with Compiled Neural Networkscited by 12
  25. 2023Stylized Data-to-Text Generation: A Case Study in the E-Commerce Domaincited by 11
  26. 2023Enhancing LLM with Evolutionary Fine Tuning for News Summary Generationcited by 10
  27. 2024LinguAlchemy: Fusing Typological and Geographical Elements for Unseen Language Generalizationcited by 10
  28. 2024CT-AGRG: Automated Abnormality-Guided Report Generation from 3D Chest CT Volumescited by 10
  29. 2022Robust Natural Language Processing: Recent Advances, Challenges, and Future Directionscited by 9
  30. 2023SentinelLMs: Encrypted Input Adaptation and Fine-tuning of Language Models for Private and Secure Inferencecited by 9
  31. 2024LLM Based Multi-Agent Generation of Semi-structured Documents from Semantic Templates in the Public Administration Domaincited by 9
  32. 2021Capturing Structural Locality in Non-parametric Language Modelscited by 8
  33. 2022Crystal Transformer: Self-learning neural language model for Generative and Tinkering Design of Materialscited by 8
  34. 2023ChatGPT-Like Large-Scale Foundation Models for Prognostics and Health Management: A Survey and Roadmapscited by 8
  35. 2024HLAT: High-quality Large Language Model Pre-trained on AWS Trainiumcited by 7
  36. 2024Towards Trustable Language Models: Investigating Information Quality of Large Language Modelscited by 6
  37. 2024Towards Lifelong Scene Graph Generation with Knowledge-ware In-context Prompt Learningcited by 6
  38. 2024A Comparative Analysis of Conversational Large Language Models in Knowledge-Based Text Generationcited by 6

50 most cited of 95 All 95 (JSON)