# Writing-RL: Advancing Long-form Writing via Adaptive Curriculum Reinforcement Learning

arXiv: https://arxiv.org/abs/2506.05760 · HTML: https://mlsanity.com/paper/2506.05760/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 20 of its 38 references that are in the map.
- First posted on arXiv 6 Jun 2025.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)

## Builds on (most cited first, 20 of 20)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training](https://mlsanity.com/paper/2501.17161/) (2025)
- [LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks](https://mlsanity.com/paper/2412.15204/) (2024)
- [Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs](https://mlsanity.com/paper/2503.01307/) (2025)
- [Inference-Time Scaling for Generalist Reward Modeling](https://mlsanity.com/paper/2504.02495/) (2025)
- [Light-R1: Curriculum SFT, DPO and RL for Long COT from Scratch and Beyond](https://mlsanity.com/paper/2503.10460/) (2025)
- [LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs](https://mlsanity.com/paper/2408.07055/) (2024)
- [Online Difficulty Filtering for Reasoning Oriented Reinforcement Learning](https://mlsanity.com/paper/2504.03380/) (2025)
- [Efficient Reinforcement Finetuning via Adaptive Curriculum Learning](https://mlsanity.com/paper/2504.05520/) (2025)
- [CritiqueLLM: Towards an Informative Critique Generation Model for Evaluation of Large Language Model Generation](https://mlsanity.com/paper/2311.18702/) (2023)
- [Guiding the Growth: Difficulty-Controllable Question Generation through Step-by-Step Rewriting](https://mlsanity.com/paper/2105.11698/) (2021)
- [Language Models can Self-Lengthen to Generate Long Texts](https://mlsanity.com/paper/2410.23933/) (2024)
- [Shifting Long-Context LLMs Research from Input to Output](https://mlsanity.com/paper/2503.04723/) (2025)
- [LongWriter-V: Enabling Ultra-Long and High-Fidelity Generation in Vision-Language Models](https://mlsanity.com/paper/2502.14834/) (2025)
- [StepMathAgent: A Step-Wise Agent for Evaluating Mathematical Processes through Tree-of-Error](https://mlsanity.com/paper/2503.10105/) (2025)

## Built on by (most cited first, 9 of 9)

- [Reinforcement Learning Foundations for Deep Research Systems: A Survey](https://mlsanity.com/paper/2509.06733/) (2025)
- [ACE-RL: Adaptive Constraint-Enhanced Reward for Long-form Generation Reinforcement Learning](https://mlsanity.com/paper/2509.04903/) (2025)
- [R2-Write: Reflection and Revision for Open-Ended Writing with Deep Reasoning](https://mlsanity.com/paper/2604.03004/) (2026)
- [DARE: Difficulty-Adaptive Reinforcement Learning with Co-Evolved Difficulty Estimation](https://mlsanity.com/paper/2605.09188/) (2026)
- [PodBench: A Comprehensive Benchmark for Instruction-Aware Audio-Oriented Podcast Script Generation](https://mlsanity.com/paper/2601.14903/) (2026)
- [POLARIS: Guiding Small Models to Write Long Stories](https://mlsanity.com/paper/2606.04095/) (2026)
- [IS-CoT: Breaking the Long-form Generation Collapse via Interleaved Structural Thinking](https://mlsanity.com/paper/2606.09709/) (2026)
- [OPERA: Aligning Open-Ended Reasoning via Objective Perplexity-based Reinforcement Learning](https://mlsanity.com/paper/2606.25757/) (2026)
- [Offline-Online Curriculum RL for Multimodal Reasoning](https://mlsanity.com/paper/2607.23700/) (2026)

## Abstract (arXiv)

Recent advances in Large Language Models(LLMs) have enabled strong performance in long-form writing, but current training paradigms remain limited: Supervised Fine-Tuning (SFT) remains constrained by data saturation and performance ceilings, while Reinforcement Learning with Verifiable Reward (RLVR), though successful in verifiable domains like math and code, cannot be directly migrated to open-ended long-form writing due to a lack of ground-truths. To further advance long-form writing, we present Writing-RL: an Adaptive Curriculum Reinforcement Learning framework to advance long-form writing capabilities beyond SFT. The framework consists of three key components: Margin-aware Data Selection strategy that prioritizes samples with high learning potential, Pairwise Comparison Reward mechanism that provides discriminative learning signals in the absence of verifiable rewards, and Dynamic Reference Scheduling approach, which plays a critical role by adaptively adjusting task difficulty based on evolving model performance. Experiments on 7B-scale writer models show that Writing-RL effectively improves long-form writing performance over strong SFT baselines. Furthermore, we observe that models trained with long-output RL generalize surprisingly well to long-input reasoning tasks, potentially offering a promising perspective for rethinking long-context training.