# PAIR: Planning and Iterative Refinement in Pre-trained Transformers for Long Text Generation

arXiv: https://arxiv.org/abs/2010.02301 · HTML: https://mlsanity.com/paper/2010.02301/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Oct 2020.

## Built on by (most cited first, 30 of 35)

- [A Survey of Controllable Text Generation using Transformer-based Pre-trained Language Models](https://mlsanity.com/paper/2201.05337/) (2022)
- [Improving Coherence and Consistency in Neural Sequence Models with Dual-System, Neuro-Symbolic Reasoning](https://mlsanity.com/paper/2107.02794/) (2021)
- [Planning with Learned Entity Prompts for Abstractive Summarization](https://mlsanity.com/paper/2104.07606/) (2021)
- [LongGenBench: Benchmarking Long-Form Generation in Long Context LLMs](https://mlsanity.com/paper/2409.02076/) (2024)
- [ORGAN: Observation-Guided Radiology Report Generation via Tree Reasoning](https://mlsanity.com/paper/2306.06466/) (2023)
- [Controllable Neural Dialogue Summarization with Personal Named Entity Planning](https://mlsanity.com/paper/2109.13070/) (2021)
- [Training Models to Generate, Recognize, and Reframe Unhelpful Thoughts](https://mlsanity.com/paper/2307.02768/) (2023)
- [PLANET: Dynamic Content Planning in Autoregressive Transformers for Long-form Text Generation](https://mlsanity.com/paper/2203.09100/) (2022)
- [PROXYQA: An Alternative Framework for Evaluating Long-Form Text Generation with Large Language Models](https://mlsanity.com/paper/2401.15042/) (2024)
- [Language modeling via stochastic processes](https://mlsanity.com/paper/2203.11370/) (2022)
- [Thinking Clearly, Talking Fast: Concept-Guided Non-Autoregressive Generation for Open-Domain Dialogue Systems](https://mlsanity.com/paper/2109.04084/) (2021)
- [Automating Steering for Safe Multimodal Large Language Models](https://mlsanity.com/paper/2507.13255/) (2025)
- [MVP: Multi-task Supervised Pre-training for Natural Language Generation](https://mlsanity.com/paper/2206.12131/) (2022)
- [Answering Ambiguous Questions via Iterative Prompting](https://mlsanity.com/paper/2307.03897/) (2023)
- [Context-Tuning: Learning Contextualized Prompts for Natural Language Generation](https://mlsanity.com/paper/2201.08670/) (2022)
- [DYPLOC: Dynamic Planning of Content Using Mixed Language Models for Text Generation](https://mlsanity.com/paper/2106.00791/) (2021)
- [To Revise or Not to Revise: Learning to Detect Improvable Claims for Argumentative Writing Support](https://mlsanity.com/paper/2305.16799/) (2023)
- [AMERICANO: Argument Generation with Discourse-driven Decomposition and Agent Interaction](https://mlsanity.com/paper/2310.20352/) (2023)
- [Automatically Summarizing Evidence from Clinical Trials: A Prototype Highlighting Current Challenges](https://mlsanity.com/paper/2303.05392/) (2023)
- [Towards Automatic Composition of ASP Programs from Natural Language Specifications](https://mlsanity.com/paper/2403.04541/) (2024)
- [Target-constrained Bidirectional Planning for Generation of Target-oriented Proactive Dialogue](https://mlsanity.com/paper/2403.06063/) (2024)
- [Sequentially Controlled Text Generation](https://mlsanity.com/paper/2301.02299/) (2023)
- [generAItor: Tree-in-the-Loop Text Generation for Language Model Explainability and Adaptation](https://mlsanity.com/paper/2403.07627/) (2024)
- [SuperWriter: Reflection-Driven Long-Form Generation with Large Language Models](https://mlsanity.com/paper/2506.04180/) (2025)
- [MOCHA: A Multi-Task Training Approach for Coherent Text Generation from Cognitive Perspective](https://mlsanity.com/paper/2210.14650/) (2022)
- [Integrating AI Planning with Natural Language Processing: A Combination of Explicit and Tacit Knowledge](https://mlsanity.com/paper/2202.07138/) (2022)
- [RSTGen: Imbuing Fine-Grained Interpretable Control into Long-FormText Generators](https://mlsanity.com/paper/2205.12590/) (2022)
- [FOCUS: Forging Originality through Contrastive Use in Self-Plagiarism for Language Models](https://mlsanity.com/paper/2406.00839/) (2024)
- [Conveying the Predicted Future to Users: A Case Study of Story Plot Prediction](https://mlsanity.com/paper/2302.09122/) (2023)
- [Guided by the Plan: Enhancing Faithful Autoregressive Text-to-Audio Generation with Guided Decoding](https://mlsanity.com/paper/2601.14304/) (2026)

## Abstract (arXiv)

Pre-trained Transformers have enabled impressive breakthroughs in generating long and fluent text, yet their outputs are often "rambling" without coherently arranged content. In this work, we present a novel content-controlled text generation framework, PAIR, with planning and iterative refinement, which is built upon a large model, BART. We first adapt the BERT model to automatically construct the content plans, consisting of keyphrase assignments and their corresponding sentence-level positions. The BART model is employed for generation without modifying its structure. We then propose a refinement algorithm to gradually enhance the generation quality within the sequence-to-sequence framework. Evaluation with automatic metrics shows that adding planning consistently improves the generation quality on three distinct domains, with an average of 20 BLEU points and 12 METEOR points improvements. In addition, human judges rate our system outputs to be more relevant and coherent than comparisons without planning.