# DPWriter: Reinforcement Learning with Diverse Planning Branching for Creative Writing

arXiv: https://arxiv.org/abs/2601.09609 · HTML: https://mlsanity.com/paper/2601.09609/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods.
- Builds on 34 of its 53 references that are in the map.
- First posted on arXiv 14 Jan 2026.

## Placed in

- Deep reinforcement learning methods / [RL training of language and agent models](https://mlsanity.com/deeprl/rl-training-of-language-and-agent-models/)

## Builds on (most cited first, 30 of 34)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models](https://mlsanity.com/paper/2506.05176/) (2025)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [Contrastive Decoding: Open-ended Text Generation as Optimization](https://mlsanity.com/paper/2210.15097/) (2022)
- [From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline](https://mlsanity.com/paper/2406.11939/) (2024)
- [Understanding the Effects of RLHF on LLM Generalisation and Diversity](https://mlsanity.com/paper/2310.06452/) (2023)
- [Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Models](https://mlsanity.com/paper/1610.02424/) (2016)
- [Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergy](https://mlsanity.com/paper/2507.01352/) (2025)
- [Does Writing with Language Models Reduce Content Diversity?](https://mlsanity.com/paper/2309.05196/) (2023)
- [Rewarding the Unlikely: Lifting GRPO Beyond Distribution Sharpening](https://mlsanity.com/paper/2506.02355/) (2025)
- [Jointly Reinforcing Diversity and Quality in Language Model Generations](https://mlsanity.com/paper/2509.02534/) (2025)
- [NoveltyBench: Evaluating Language Models for Humanlike Diversity](https://mlsanity.com/paper/2504.05228/) (2025)
- [TreePO: Bridging the Gap of Policy Optimization and Efficacy and Inference Efficiency with Heuristic Tree-based Modeling](https://mlsanity.com/paper/2508.17445/) (2025)
- [Evaluating the Diversity and Quality of LLM Generated Content](https://mlsanity.com/paper/2504.12522/) (2025)
- [Modifying Large Language Model Post-Training for Diverse Creative Writing](https://mlsanity.com/paper/2503.17126/) (2025)
- [ParaThinker: Native Parallel Thinking as a New Paradigm to Scale LLM Test-time Compute](https://mlsanity.com/paper/2509.04475/) (2025)
- [ETTRL: Balancing Exploration and Exploitation in LLM Test-Time Reinforcement Learning Via Entropy Mechanism](https://mlsanity.com/paper/2508.11356/) (2025)
- [Reverse-Engineered Reasoning for Open-Ended Generation](https://mlsanity.com/paper/2509.06160/) (2025)
- [Rethinking and Refining the Distinct Metric](https://mlsanity.com/paper/2202.13587/) (2022)
- [Diverse Preference Learning for Capabilities and Alignment](https://mlsanity.com/paper/2511.08594/) (2025)
- [Language Models that Think, Chat Better](https://mlsanity.com/paper/2509.20357/) (2025)
- [Post-training Large Language Models for Diverse High-Quality Responses](https://mlsanity.com/paper/2509.04784/) (2025)
- [Representation-Based Exploration for Language Models: From Test-Time to Post-Training](https://mlsanity.com/paper/2510.11686/) (2025)
- [G2: Guided Generation for Enhanced Output Diversity in LLMs](https://mlsanity.com/paper/2511.00432/) (2025)
- [Igniting Creative Writing in Small Language Models: LLM-as-a-Judge versus Multi-Agent Refined Rewards](https://mlsanity.com/paper/2508.21476/) (2025)
- [Evaluating Text Creativity across Diverse Domains: A Dataset and Large Language Model Evaluator](https://mlsanity.com/paper/2505.19236/) (2025)
- [Diverse, not Short: A Length-Controlled Data Selection Strategy for Improving Response Diversity of Language Models](https://mlsanity.com/paper/2505.16245/) (2025)

## Built on by (most cited first, 6 of 6)

- [CreativeBench: Benchmarking and Enhancing Machine Creativity via Self-Evolving Challenges](https://mlsanity.com/paper/2603.11863/) (2026)
- [POLARIS: Guiding Small Models to Write Long Stories](https://mlsanity.com/paper/2606.04095/) (2026)
- [OPERA: Aligning Open-Ended Reasoning via Objective Perplexity-based Reinforcement Learning](https://mlsanity.com/paper/2606.25757/) (2026)
- [StorySpark: Module-wise Evolutionary Search for Story Premise Generation](https://mlsanity.com/paper/2608.12336/) (2026)
- [Strategically Diverse Sampling for Self-Training](https://mlsanity.com/paper/2609.31571/) (2026)
- [LLMs as Adaptive Meta-Solvers: Strategy-Diverse RL for Industrial-Scale Optimization](https://mlsanity.com/paper/2609.34427/) (2026)

## Abstract (arXiv)

Reinforcement learning (RL)-based enhancement of large language models (LLMs) often leads to reduced output diversity, undermining their utility in open-ended tasks like creative writing. Current methods lack explicit mechanisms for guiding diverse exploration and instead prioritize optimization efficiency and performance over diversity. This paper proposes an RL framework structured around a semi-structured long Chain-of-Thought (CoT), in which the generation process is decomposed into explicitly planned intermediate steps. We introduce a Diverse Planning Branching method that strategically introduces divergence at the planning phase based on diversity variation, alongside a group-aware diversity reward to encourage distinct trajectories. Experimental results on creative writing benchmarks demonstrate that our approach significantly improves output diversity without compromising generation quality, consistently outperforming existing baselines.