R2PO: Decoupling Rollout and Inference Policies for LLM Reasoning
On the map: LLM reasoning → Reinforcement learning for reasoning
Abstract (arXiv)
Existing reinforcement learning methods for LLM reasoning implicitly assume that the policy generating training trajectories should coincide with the one producing inference responses. We argue that this is a misleading inductive bias: the optimization-optimal trajectory distribution favors informative gradients, whereas the inference-optimal response distribution emphasizes accuracy and consistency. Forcing both into a single policy entangles their gradients and suppresses exploration. We propose R2PO (Residual Rollout Policy Optimization), which attaches a lightweight Residual Rollout-Head atop the policy to decouple training trajectories from inference responses, diversifying rollouts during training while keeping inference generation intact. Experiments show that R2PO consistently outperforms baselines, with average accuracy gains of 3.4% on MATH-500 and 1.3% on APPS, alongside more diverse rollouts and reduced length bias. Our code is available at https://github.com/RRPO-ARR/Code.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 17 of its 36 references that are in the map. First posted on arXiv 17 Jan 2026.
Builds on
17 of 36 references.
Closest ancestors · 7
Same direction, newest first
- 2025Clip-Low Increases Entropy and Clip-High Decreases Entropy in Reinforcement Learning of Large Language Models30 Sep
- 2025Reasoning with Exploration: An Entropy Perspective17 Jun
- 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
- 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
- 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
- 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Training22 Nov
- 2017Proximal Policy Optimization Algorithms20 Jul
Other references · 10
- 2021Training Verifiers to Solve Math Word Problemscited by 9,743
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
- 2025Qwen3 Technical Reportcited by 8,195
- 2021Evaluating Large Language Models Trained on Codecited by 8,053
- 2024Qwen2.5 Technical Reportcited by 4,713
- 2021Program Synthesis with Large Language Modelscited by 3,509
- 2024DeepSeek-V3 Technical Reportcited by 724
- 2025ReMA: Learning to Meta-think for LLMs with Multi-Agent Reinforcement Learningcited by 78
- 2025EPO: Entropy-regularized Policy Optimization for LLM Agents Reinforcement Learningcited by 15
- 2024Entropy-Regularized Token-Level Policy Optimization for Language Agent Reinforcementcited by 12
Built on by
Citing papers per year: 2026: 3.
3 papers.
Similar, not cited
Close in wording, no citation either way
- 2026Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learning
- 2025Trajectory Bellman Residual Minimization: A Simple Value-Based Method for LLM Reasoning
- 2025Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts
- 2025Accelerating RL for LLM Reasoning with Optimal Advantage Regression
- 2026Learning as Reasoning Unfolds: Progressive Rollout Allocation for Efficient Reinforcement Learning
- 2026BASIS: Batchwise Advantage Estimation from Single-Rollout Information Sharing for LLM Reasoning
- 2026Reusing Rollouts under Policy Lag: Prefix-Normalized Policy Optimization for LLM Reinforcement Learning
- 2025Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning