# Progressive Generation of Long Text with Pretrained Language Models

arXiv: https://arxiv.org/abs/2006.15720 · HTML: https://mlsanity.com/paper/2006.15720/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Jun 2020.

## Built on by (most cited first, 14 of 14)

- [Long Text Generation by Modeling Sentence-Level and Discourse-Level Coherence](https://mlsanity.com/paper/2105.08963/) (2021)
- [Progressive Transformer-Based Generation of Radiology Reports](https://mlsanity.com/paper/2102.09777/) (2021)
- [Parallel Refinements for Lexically Constrained Text Generation with BART](https://mlsanity.com/paper/2109.12487/) (2021)
- [GraphPlan: Story Generation by Planning with Event Graph](https://mlsanity.com/paper/2102.02977/) (2021)
- [A Comprehensive Survey of Mamba Architectures for Medical Image Analysis: Classification, Segmentation, Restoration and Beyond](https://mlsanity.com/paper/2410.02362/) (2024)
- [InsNet: An Efficient, Flexible, and Performant Insertion-based Text Generation Model](https://mlsanity.com/paper/2102.11008/) (2021)
- [DiscoDVT: Generating Long Text with Discourse-Aware Discrete Variational Transformer](https://mlsanity.com/paper/2110.05999/) (2021)
- [LongStory: Coherent, Complete and Length Controlled Long story Generation](https://mlsanity.com/paper/2311.15208/) (2023)
- [Changing the Mind of Transformers for Topically-Controllable Language Generation](https://mlsanity.com/paper/2103.15335/) (2021)
- [Advancing Precise Outline-Conditioned Text Generation with Task Duality and Explicit Outline Control](https://mlsanity.com/paper/2305.14459/) (2023)
- [Semantic Frame Forecast](https://mlsanity.com/paper/2104.05604/) (2021)
- [BERT Goes Off-Topic: Investigating the Domain Transfer Challenge using Genre Classification](https://mlsanity.com/paper/2311.16083/) (2023)
- [Sentence Semantic Regression for Text Generation](https://mlsanity.com/paper/2108.02984/) (2021)
- [Goal-driven text descriptions for images](https://mlsanity.com/paper/2108.12575/) (2021)

## Abstract (arXiv)

Large-scale language models (LMs) pretrained on massive corpora of text, such as GPT-2, are powerful open-domain text generators. However, as our systematic examination reveals, it is still challenging for such models to generate coherent long passages of text (e.g., 1000 tokens), especially when the models are fine-tuned to the target domain on a small corpus. Previous planning-then-generation methods also fall short of producing such long text in various domains. To overcome the limitations, we propose a simple but effective method of generating text in a progressive manner, inspired by generating images from low to high resolution. Our method first produces domain-specific content keywords and then progressively refines them into complete passages in multiple stages. The simple design allows our approach to take advantage of pretrained LMs at each stage and effectively adapt to any target domain given only a small set of examples. We conduct a comprehensive empirical study with a broad set of evaluation metrics, and show that our approach significantly improves upon the fine-tuned large LMs and various planning-then-generation methods in terms of quality and sample efficiency. Human evaluation also validates that our model generations are more coherent.