mlsanityData to 7 Oct 2026

Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learning

arXiv 2605.02913 ↗First posted 8 Apr 2026cs.LGcited by 8

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning (RL) has become a central post-training tool for improving the reasoning abilities of large language models (LLMs). In these systems, the rollout, the trajectory sampled from a prompt to termination, including intermediate reasoning steps and optional tool or environment interactions, determines the data the optimizer learns from, yet rollout design is often underreported. This survey provides an optimizer-agnostic view of rollout strategies for RL-based post-training of reasoning LLMs. We formalize rollout pipelines with unified notation and introduce Generate-Filter-Control-Replay (GFCR), a lifecycle taxonomy that decomposes rollout pipelines into four modular stages: Generate proposes candidate trajectories and topologies; Filter constructs intermediate signals via verifiers, judges, critics; Control allocates compute and makes continuation/branching/stopping decisions under budgets; and Replay retains and reuses artifacts across rollouts without weight updates, including self-evolving curricula that autonomously generate new training tasks. We complement GFCR with a criterion taxonomy of reliability, coverage, and cost sensitivity that characterizes rollout trade-offs. Using this framework, we synthesize methods spanning RL with verifiable rewards, process supervision, judge-based gating, guided and tree/segment rollouts, adaptive compute allocation, early-exit and partial rollouts, throughput optimization, and replay/recomposition for self-improvement. We ground the framework with case studies in math, code/SQL, multimodal reasoning, tool-using agents, and agentic skill benchmarks that evaluate skill induction, reuse, and cross-task transfer. Finally, we provide a diagnostic index that maps common rollout pathologies to GFCR modules and mitigation levers, alongside open challenges for building reproducible, compute-efficient, and trustworthy rollout pipelines.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 159 of its 178 references that are in the map. First posted on arXiv 8 Apr 2026.

Builds on

159 of 178 references.

Closest ancestors · 59

Same direction, newest first

  1. 2026WS-GRPO: Weakly-Supervised Group-Relative Policy Optimization for Rollout-Efficient Reasoning19 Feb
  2. 2025TreeGRPO: Tree-Advantage GRPO for Online RL Post-Training of Diffusion Models9 Dec
  3. 2025VADE: Variance-Aware Dynamic Sampling via Online Sample-Level Difficulty Estimation for Multimodal RL24 Nov
  4. 2025Taming the Long-Tail: Efficient Reasoning RL Training with Adaptive Drafter20 Nov
  5. 2025Zero Reinforcement Learning Towards General Domains29 Oct
  6. 2025RAVR: Reference-Answer-guided Variational Reasoning for Large Language Models29 Oct
  7. 2025Lookahead Tree-Based Rollouts for Enhanced Trajectory-Level Exploration in Reinforcement Learning with Verifiable Rewards28 Oct
  8. 2025Scaf-GRPO: Scaffolded Group Relative Policy Optimization for Enhancing LLM Reasoning22 Oct
  9. 2025Reasoning with Sampling: Your Base Model is Smarter Than You Think16 Oct
  10. 2025XRPO: Pushing the limits of GRPO with Targeted Exploration and Exploitation8 Oct
  11. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Survey2 Oct
  12. 2025Plan Then Action:High-Level Planning Guidance Reinforcement Learning for LLM Reasoning2 Oct
38 more
  1. 2025RoRecomp: Enhancing Reasoning Efficiency via Rollout Response Recomposition in Reinforcement Learning30 Sep
  2. 2025Improving Sampling Efficiency in RLVR through Adaptive Rollout and Response Reuse30 Sep
  3. 2025DeepSearch: Overcome the Bottleneck of Reinforcement Learning with Verifiable Rewards via Monte Carlo Tree Search29 Sep
  4. 2025A State-of-the-Art SQL Reasoning Model using RLVR25 Sep
  5. 2025VCRL: Variance-based Curriculum Reinforcement Learning for Large Language Models24 Sep
  6. 2025Exploiting Tree Structure for Credit Assignment in RL Training of LLMs22 Sep
  7. 2025Tree-OPO: Off-policy Monte Carlo Tree-Guided Advantage Optimization for Multistep Reasoning11 Sep
  8. 2025A Survey of Reinforcement Learning for Large Reasoning Models10 Sep
  9. 2025TreePO: Bridging the Gap of Policy Optimization and Efficacy and Inference Efficiency with Heuristic Tree-based Modeling24 Aug
  10. 2025G2RPO-A: Guided Group Relative Policy Optimization with Adaptive Guidance18 Aug
  11. 2025Sample More to Think Less: Group Filtered Policy Optimization for Concise Reasoning13 Aug
  12. 2025Squeeze the Soaked Sponge: Efficient Off-policy Reinforcement Finetuning for Large Language Model9 Jul
  13. 2025From Data-Centric to Sample-Centric: Enhancing LLM Reasoning via Progressive Optimization9 Jul
  14. 2025A Technical Survey of Reinforcement Learning Techniques for Large Language Models5 Jul
  15. 2025ReasonFlux-PRM: Trajectory-Aware PRMs for Long Chain-of-Thought Reasoning in LLMs23 Jun
  16. 2025RLPR: Extrapolating RLVR to General Domains without Verifiers23 Jun
  17. 2025BREAD: Branched Rollouts from Expert Anchors Bridge SFT & RL for Reasoning20 Jun
  18. 2025Spurious Rewards: Rethinking Training Signals in RLVR12 Jun
  19. 2025Improving Data Efficiency for LLM Reinforcement Fine-tuning Through Difficulty-targeted Online Data Selection and Rollout Replay5 Jun
  20. 2025TreeRPO: Tree Relative Policy Optimization5 Jun
  21. 2025Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts2 Jun
  22. 2025From Accuracy to Robustness: A Study of Rule- and Model-based Verifiers in Mathematical Reasoning28 May
  23. 2025Reinforcing General Reasoning without Verifiers27 May
  24. 2025VerifyBench: Benchmarking Reference-based Reward Systems for Large Language Models21 May
  25. 2025Self-Evolving Curriculum for LLM Reasoning20 May
  26. 2025SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy Optimization18 May
  27. 2025Stepwise Guided Policy Optimization: Coloring your Incorrect Reasoning in GRPO16 May
  28. 2025Group-in-Group Policy Optimization for LLM Agent Training16 May
  29. 2025S-GRPO: Early Exit via Reinforcement Learning in Reasoning Models12 May
  30. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Example29 Apr
  31. 2025Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning18 Apr
  32. 2025GRPO-LEAD: A Difficulty-Aware Reinforcement Learning Approach for Concise Mathematical Reasoning in Language Models13 Apr
  33. 2025GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoning1 Apr
  34. 2025Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across Diverse Domains31 Mar
  35. 2025Trajectory Balance with Asynchrony: Decoupling Exploration and Learning for Fast, Scalable LLM Post-Training24 Mar
  36. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  37. 2025R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimization17 Mar
  38. 2025Reinforcement Learning with Verifiable Rewards: GRPO's Effective Loss, Dynamics, and Success Amplification9 Mar

50 newest of 59 All 59 (JSON)

Other references · 100

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  5. 2017Deep reinforcement learning from human preferencescited by 3,846
  6. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  7. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  8. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  9. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,628
  10. 2024A Survey on LLM-as-a-Judgecited by 1,384
  11. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  12. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
38 more
  1. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  2. 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 667
  3. 2025Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Modelscited by 653
  4. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  5. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 580
  6. 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Modelcited by 515
  7. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  8. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 357
  9. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 342
  10. 2024Agent Workflow Memorycited by 342
  11. 2024Training Software Engineering Agents and Verifiers with SWE-Gymcited by 300
  12. 2023A Survey of Reinforcement Learning from Human Feedbackcited by 259
  13. 2026SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learningcited by 252
  14. 2024Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judgecited by 249
  15. 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learningcited by 243
  16. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 195
  17. 2025Generalizing Verifiable Instruction Followingcited by 178
  18. 2025SkillWeaver: Web Agents can Self-Improve by Discovering and Honing Skillscited by 174
  19. 2024The BrowserGym Ecosystem for Web Agent Researchcited by 152
  20. 2025SpaceR: Reinforcing MLLMs in Video Spatial Reasoningcited by 152
  21. 2025AdaptThink: Reasoning Models Can Learn When to Thinkcited by 141
  22. 2025AbstentionBench: Reasoning LLMs Fail on Unanswerable Questionscited by 129
  23. 2025EvolveR: Self-Evolving LLM Agents through an Experience-Driven Lifecyclecited by 129
  24. 2026MemSkill: Learning and Evolving Memory Skills for Self-Evolving Agentscited by 129
  25. 2025From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3Dcited by 127
  26. 2025Inducing Programmatic Skills for Agentic Taskscited by 103
  27. 2024GUI Agents: A Surveycited by 101
  28. 2025VerlTool: Towards Holistic Agentic Reinforcement Learning with Tool Usecited by 92
  29. 2025Reinforcement Learning for Self-Improving Agent with Skill Librarycited by 91
  30. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 89
  31. 2025TreeRL: LLM Reinforcement Learning with On-Policy Tree Searchcited by 75
  32. 2025Tree Search for LLM Agent Reinforcement Learningcited by 73
  33. 2025ShorterBetter: Guiding Reasoning Models to Find Optimal Inference Length for Efficient Reasoningcited by 70
  34. 2025Agent0: Unleashing Self-Evolving Agents from Zero Data via Tool-Integrated Reasoningcited by 68
  35. 2025Think Only When You Need with Large Hybrid-Reasoning Modelscited by 64
  36. 2025BranchGRPO: Stable and Efficient GRPO with Structured Branching in Diffusion Modelscited by 59
  37. 2025xVerify: Efficient Answer Verifier for Reasoning Model Evaluationscited by 53
  38. 2024Visual Prompting in Multimodal Large Language Models: A Surveycited by 52

50 most cited of 100 All 100 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 8.

8 papers.

  1. 2026OLIVIA: Online Learning via Inference-time Action Adaptation for Decision Making in LLM ReAct Agentscited by 2
  2. 2026MASS-DPO: Multi-negative Active Sample Selection for Direct Policy Optimizationcited by 1
  3. 2026F-GRPO: Factorized Group-Relative Policy Optimization for Unified Candidate Generation and Rankingcited by 1
  4. 2026SimpleSearch-VL: A Simple Recipe for Multimodal Agentic Deep Searchcited by 1
  5. 2026Tandem Reinforcement Learning with Verifiable Rewardscited by 0
  6. 2026Can We Break LLMs Out of Self-Loops? Fine-Grained Reasoning Control with Activation Steeringcited by 0
  7. 2026RRPO: Reference-Relative Policy Optimization with Stratified Conditional Rolloutscited by 0
  8. 2026Rollout Efficiency in Reinforcement Learning for Reasoning Large Language Models: A Taxonomy and Future Directionscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Improving RL Exploration for LLM Reasoning through Retrospective Replay
  2. 2026R2PO: Decoupling Rollout and Inference Policies for LLM Reasoning
  3. 2026RolloutPipe: Overlapping Pipelined Rollout and Training in Disaggregated On-Policy LLM Reinforcement Learning
  4. 2025RollPacker: Mitigating Long-Tail Rollouts for Fast, Synchronous RL Post-Training
  5. 2026BASIS: Batchwise Advantage Estimation from Single-Rollout Information Sharing for LLM Reasoning
  6. 2025Accelerating RL for LLM Reasoning with Optimal Advantage Regression
  7. 2026Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning
  8. 2025Enhancing Efficiency and Exploration in Reinforcement Learning for LLMs