mlsanityData to 7 Oct 2026

Pretrained Language Models for Text Generation: A Survey

arXiv 2201.05273 ↗First posted 14 Jan 2022cs.CLcited by 121

Not on the map. Why?

Abstract (arXiv)

Text Generation aims to produce plausible and readable text in a human language from input data. The resurgence of deep learning has greatly advanced this field, in particular, with the help of neural generation models based on pre-trained language models (PLMs). Text generation based on PLMs is viewed as a promising approach in both academia and industry. In this paper, we provide a survey on the utilization of PLMs in text generation. We begin with introducing three key aspects of applying PLMs to text generation: 1) how to encode the input into representations preserving input semantics which can be fused into PLMs; 2) how to design an effective PLM to serve as the generation model; and 3) how to effectively optimize PLMs given the reference text and to ensure that the generated texts satisfy special text properties. Then, we show the major challenges arisen in these aspects, as well as possible solutions for them. We also include a summary of various useful resources and typical text generation applications based on PLMs. Finally, we highlight the future research directions which will further improve these PLMs for text generation. This comprehensive survey is intended to help researchers interested in text generation problems to learn the core concepts, the main techniques and the latest developments in this area based on PLMs.

Key facts. Cited by 121 papers in the map (data through 7 Oct 2026). Builds on 224 of its 324 references that are in the map. First posted on arXiv 14 Jan 2022.

Builds on

224 of 324 references.

References · 224

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2017Proximal Policy Optimization Algorithmscited by 14,441
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2020Scaling Laws for Neural Language Modelscited by 6,411
  9. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  10. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  11. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  12. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
38 more
  1. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  2. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  3. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  4. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  6. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  7. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  8. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  9. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  10. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  11. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  12. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  13. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  14. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  15. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  16. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  17. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  18. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  19. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  20. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  21. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  22. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
  23. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  24. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  25. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  26. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  27. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  28. 2018Hierarchical Neural Story Generationcited by 1,182
  29. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  30. 2017Mixed Precision Trainingcited by 1,135
  31. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  32. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  33. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  34. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  35. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  36. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  37. 2018Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labelscited by 994
  38. 2019How Can We Know What Language Models Know?cited by 953

50 most cited of 224 All 224 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 1; 2024: 45; 2025: 57; 2026: 17.

121 papers.

Sort:

By area:

  1. 2023A Survey on Uncertainty Quantification Methods for Deep Learningcited by 78
  2. 2024Foundation Model for Advancing Healthcare: Challenges, Opportunities, and Future Directionscited by 77
  3. 2024Machine Unlearning in Generative AI: A Surveycited by 49
  4. 2024Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologiescited by 44
  5. 2024Natural Language Processing for the Legal Domain: A Survey of Tasks, Datasets, Models, and Challengescited by 42
  6. 2024Large Language Model Inference Acceleration: A Comprehensive Hardware Perspectivecited by 36
  7. 2024Generative AI for Synthetic Data Generation: Methods, Challenges and the Futurecited by 31
  8. 2024A Systematic Review on Long-Tailed Learningcited by 27
  9. 2025Retrieval Augmented Generation with Collaborative Filtering for Personalized Text Generationcited by 23
  10. 2024The Price of Prompting: Profiling Energy Use in Large Language Models Inferencecited by 22
  11. 2024Bileve: Securing Text Provenance in Large Language Models Against Spoofing with Bi-level Signaturecited by 21
  12. 2024Unlocking Memorization in Large Language Models with Dynamic Soft Promptingcited by 21
38 more
  1. 2025Exploring LLM Agents for Cleaning Tabular Machine Learning Datasetscited by 20
  2. 2024DiffZOO: A Purely Query-Based Black-Box Attack for Red-teaming Text-to-Image Generative Model via Zeroth Order Optimizationcited by 19
  3. 2024Text Generation: A Systematic Literature Review of Tasks, Evaluation, and Challengescited by 18
  4. 2025A Survey of AIOps in the Era of Large Language Modelscited by 18
  5. 2025Large Language Models for Computer-Aided Design: A Surveycited by 16
  6. 2024Contextualized Counterspeech: Strategies for Adaptation, Personalization, and Evaluationcited by 15
  7. 2025Empowering Time Series Analysis with Synthetic Data: A Survey and Outlook in the Era of Foundation Modelscited by 15
  8. 2025Mark Your LLM: Detecting the Misuse of Open-Source Large Language Models via Watermarkingcited by 14
  9. 2025Pre-Trained Policy Discriminators are General Reward Modelscited by 14
  10. 2025A Survey of Attacks on Large Language Modelscited by 13
  11. 2025UniGuardian: A Unified Defense for Detecting Prompt Injection, Backdoor Attacks and Adversarial Attacks in Large Language Modelscited by 12
  12. 2024Learn and Unlearn: Addressing Misinformation in Multilingual LLMscited by 11
  13. 2024Can LLMs be Good Graph Judge for Knowledge Graph Construction?cited by 11
  14. 2024GPT for Games: A Scoping Review (2020-2023)cited by 10
  15. 2024Fairness Definitions in Language Models Explainedcited by 10
  16. 2024Adaptive Pre-training Data Detection for Large Language Models via Surprising Tokenscited by 10
  17. 2024GPT for Games: An Updated Scoping Review (2020-2024)cited by 10
  18. 2025ELBA-Bench: An Efficient Learning Backdoor Attacks Benchmark for Large Language Modelscited by 10
  19. 2024Multi-Bin Batching for Increasing LLM Inference Throughputcited by 9
  20. 2024Enhancing Table Recognition with Vision LLMs: A Benchmark and Neighbor-Guided Toolchain Reasonercited by 9
  21. 2025From Text to Trust: Empowering AI-assisted Decision Making with Adaptive LLM-powered Analysiscited by 9
  22. 2024The GPT-WritingPrompts Dataset: A Comparative Analysis of Character Portrayal in Short Storiescited by 8
  23. 2024Adaptive Draft-Verification for Efficient Large Language Model Decodingcited by 8
  24. 2025WhatELSE: Shaping Narrative Spaces at Configurable Level of Abstraction for AI-bridged Interactive Storytellingcited by 8
  25. 2025Guardians and Offenders: A Survey on Harmful Content Generation and Safety Mitigation of LLMcited by 8
  26. 2024What Is Missing in Multilingual Visual Reasoning and How to Fix Itcited by 7
  27. 2024Integrating Planning into Single-Turn Long-Form Text Generationcited by 7
  28. 2024Inevitable Trade-off between Watermark Strength and Speculative Sampling Efficiency for Language Modelscited by 7
  29. 2025Consistency of Responses and Continuations Generated by Large Language Models on Social Mediacited by 7
  30. 2025NutriGen: Personalized Meal Plan Generator Leveraging Large Language Models to Enhance Dietary and Nutritional Adherencecited by 7
  31. 2025Enhancing Decision-Making of Large Language Models via Actor-Criticcited by 7
  32. 2024Large Language Model Enhanced Knowledge Representation Learning: A Surveycited by 6
  33. 2025Survey and Experiments on Mental Disorder Detection via Social Media: From Large Language Models and RAG to Agentscited by 6
  34. 2025Layer as Puzzle Pieces: Compressing Large Language Models through Layer Concatenationcited by 6
  35. 2025Probe-Free Low-Rank Activation Interventioncited by 5
  36. 2025A Novel Hat-Shaped Device-Cloud Collaborative Inference Framework for Large Language Modelscited by 5
  37. 2025CoRank: LLM-Based Compact Reranking with Document Features for Scientific Retrievalcited by 5
  38. 2025BucketServe: Bucket-Based Dynamic Batching for Smart and Efficient LLM Inference Servingcited by 5

50 most cited of 121 All 121 (JSON)