mlsanityData to 14 Oct 2026

Rollout Efficiency in Reinforcement Learning for Reasoning Large Language Models: A Taxonomy and Future Directions

arXiv 2609.25463 ↗First posted 21 Sep 2026cs.AI cs.DCcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reasoning-oriented reinforcement learning enables large language models to solve mathematical, coding, and other multi-step tasks, but shifts a substantial portion of the training cost to rollout, where trajectories are generated for policy updates. Efficient rollout mechanisms are therefore essential to reduce this cost while maintaining the freshness, consistency, and statistical validity of training data. This survey provides a systematic taxonomy of recent research on rollout efficiency for reasoning-oriented reinforcement learning, classifying existing approaches from both mechanism and bottleneck perspectives. Based on this taxonomy, we analyze how different technique families address distinct sources of rollout inefficiency, examine opportunities and potential conflicts for combining them, identify gaps in the evaluation and reporting of efficiency gains, and discuss open challenges and future research directions.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 92 of its 124 references that are in the map. First posted on arXiv 21 Sep 2026.

Builds on

92 of 124 references.

Closest ancestors · 37

Same direction, newest first

  1. 2026Headroom-Drift Replay: A Primitive for Principled Replay Control in GRPO3 Sep
  2. 2026Beyond Solvability: Task Learnability as a Static Prior for LLM RL Post-Training10 Aug
  3. 2026SpecRoll: Fast-Slow Verifier-Feedback Adaptation for Speculative Reinforcement Learning Rollouts5 Aug
  4. 2026LEEPS: Latent-Guided Explore-Exploit Prompt Sampling for Efficient RLVR in Large Language Models30 Jul
  5. 2026Kalman Meets Curriculum: Efficient Dynamic Prompt Selection for Adaptive RL Finetuning30 Jul
  6. 2026Learning as Reasoning Unfolds: Progressive Rollout Allocation for Efficient Reinforcement Learning24 Jul
  7. 2026RolloutPipe: Overlapping Pipelined Rollout and Training in Disaggregated On-Policy LLM Reinforcement Learning25 Jun
  8. 2026EfficientRollout: System-Aware Self-Speculative Decoding for RL Rollouts17 Jun
  9. 2026Sparrow: Sparse Rollout for Stable and Efficient Long-context RL of Large Language Models7 Jun
  10. 2026Rollout-Level Advantage-Prioritized Experience Replay for GRPO3 Jun
  11. 2026RLVR without Ineffective Samples: Group Prioritized Off-Policy Optimization for LLM Reasoning31 May
  12. 2026ESPO: Early-Stopping Proximal Policy Optimization28 May
25 more
  1. 2026How Off-Policy Can GRPO Be? Mu-GRPO for Efficient LLM Reinforcement Learning17 May
  2. 2026Where to Spend Rollouts: Hit-Utility Optimal Rollout Allocation for Group-Based RLVR8 May
  3. 2026How to Compress KV Cache in RL Post-Training? Shadow Mask Distillation for Memory-Efficient Alignment7 May
  4. 2026Selective Rollout: Mid-Trajectory Termination for Multi-Sample Agent RL7 May
  5. 2026From Reasoning to Agentic: Credit Assignment in Reinforcement Learning for Large Language Models10 Apr
  6. 2026Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learning8 Apr
  7. 2026Train at Moving Edge: Online-Verified Prompt Selection for Efficient RL Training of Large Reasoning Model26 Mar
  8. 2026SortedRL: Accelerating RL Training for LLMs through Online Length-Aware Scheduling24 Mar
  9. 2026Train Less, Learn More: Adaptive Efficient Rollout Optimization for Group-Based Reinforcement Learning15 Feb
  10. 2026Difficulty-Estimated Policy Optimization6 Feb
  11. 2026HeaPA: Difficulty-Aware Heap Sampling and On-Policy Query Augmentation for LLM Reinforcement Learning30 Jan
  12. 2026Jet-RL: Enabling On-Policy FP8 Reinforcement Learning with Unified Training and Rollout Precision Flow20 Jan
  13. 2026Sparse-RL: Breaking the Memory Wall in LLM Reinforcement Learning via Stable Sparse Rollouts15 Jan
  14. 2025RLBoost: Harvesting Preemptible Resources for Cost-Efficient Reinforcement Learning on LLMs22 Oct
  15. 2025SPEC-RL: Accelerating On-Policy Reinforcement Learning with Speculative Rollouts27 Sep
  16. 2025RollPacker: Mitigating Long-Tail Rollouts for Fast, Synchronous RL Post-Training25 Sep
  17. 2025VCRL: Variance-based Curriculum Reinforcement Learning for Large Language Models24 Sep
  18. 2025A Survey of Reinforcement Learning for Large Reasoning Models10 Sep
  19. 2025History Rhymes: Accelerating LLM Reinforcement Learning with RhymeRL26 Aug
  20. 2025Reinforcing Multi-Turn Reasoning in LLM Agents via Fine-Grained Reward Structure and Credit Assignment17 May
  21. 2025Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning18 Apr
  22. 2025Agentic Reward Modeling: Integrating Human Preferences with Verifiable Correctness Signals for Reliable Reward Systems26 Feb
  23. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  24. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  25. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 55

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  3. 2024OpenAI o1 System Cardcited by 1,244
  4. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  5. 2024Self-Rewarding Language Modelscited by 681
  6. 2023Reward Design with Language Modelscited by 261
  7. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 135
  8. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 132
  9. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 116
  10. 2024Rule Based Rewards for Language Model Safetycited by 108
  11. 2015Massively Parallel Methods for Deep Reinforcement Learningcited by 82
  12. 2025Efficient Inference for Large Reasoning Models: A Surveycited by 61
38 more
  1. 2025StreamRL: Scalable, Heterogeneous, and Elastic RL for LLMs with Disaggregated Stream Generationcited by 61
  2. 2025A Survey on Post-training of Large Language Modelscited by 54
  3. 2025LlamaRL: A Distributed Asynchronous Reinforcement Learning Framework for Efficient Large-scale LLM Trainingcited by 42
  4. 2025AsyncFlow: An Asynchronous Streaming RL Framework for Efficient LLM Post-Trainingcited by 41
  5. 2026Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learningcited by 37
  6. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecyclecited by 35
  7. 2025Generalist Reward Models: Found Inside Large Language Modelscited by 34
  8. 2025APRIL: Active Partial Rollouts in Reinforcement Learning to Tame Long-tail Generationcited by 30
  9. 2024Efficient Training of Large Language Models on Distributed Infrastructures: A Surveycited by 27
  10. 2026ProRL Agent: Rollout-as-a-Service for RL Training of Multi-Turn LLM Agentscited by 20
  11. 2026Are Full Rollouts Necessary for On-Policy Distillation?cited by 17
  12. 2026DORA: A Scalable Asynchronous Reinforcement Learning System for Language Model Trainingcited by 14
  13. 2026FP8-RL: A Practical and Stable Low-Precision Stack for LLM Reinforcement Learningcited by 13
  14. 2026StaleFlow: Staleness-Aware Data Management for Mitigating Data Skewness in Fully Disaggregated RL Post-Trainingcited by 11
  15. 2025DistFlow: A Fully Distributed RL Framework for Scalable and Efficient LLM Post-Trainingcited by 10
  16. 2026TRACE: A Unified Rollout Budget Allocation Framework for Efficient Agentic Reinforcement Learningcited by 10
  17. 2025AReaL-Hex: Accommodating Asynchronous RL Training over Heterogeneous GPUscited by 9
  18. 2026Accelerating RL Post-Training Rollouts via System-Integrated Speculative Decodingcited by 9
  19. 2026Actor-Curator: Co-adaptive Curriculum Learning via Policy-Improvement Bandits for RL Post-Trainingcited by 8
  20. 2026Heddle: A Distributed Orchestration System for Agentic RL Rolloutcited by 8
  21. 2026ROSE: Rollout On Serving GPUs via Cooperative Elasticity for Agentic RLcited by 8
  22. 2026Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Applicationcited by 8
  23. 2026QaRL: Rollout-Aligned Quantization-Aware RL for Fast and Stable Training under Training--Inference Mismatchcited by 7
  24. 2026Relax: An Asynchronous Reinforcement Learning Engine for Omni-Modal Post-Training at Scalecited by 7
  25. 2026Diagnosing Training Inference Mismatch in LLM Reinforcement Learning via a Zero-Mismatch Referencecited by 7
  26. 2026Bidirectional Resource Scheduling for Disaggregated and Asynchronous RL Post-Trainingcited by 6
  27. 2026Spend Your Rollouts Where It Counts: Rollout Allocation for Group-Based RL Post-Trainingcited by 5
  28. 2026Learning from the Right Rollouts: Data Attribution for PPO-based LLM Post-Trainingcited by 4
  29. 2026Libra: Efficient Resource Management for Agentic RL Post-Trainingcited by 4
  30. 2026TideRL: Boosting Agentic RL Goodput with Readiness-Aware Schedulingcited by 4
  31. 2026DUET: Optimize Token-Budget Allocation for Reinforcement Learning with Verifiable Rewardscited by 3
  32. 2026AIS: Adaptive Importance Sampling for Quantized RLcited by 3
  33. 2026WAR: Workload-Aware Rollouts for Synchronous Agentic Reinforcement Learningcited by 3
  34. 2026AReaL-DTE: Sparse Policy-Weight Transfer for Online Agentic Reinforcement Learningcited by 3
  35. 2026Online Draft Co-Training for Speculative Decoding in Large-Scale, Long-Context RL Post-Trainingcited by 3
  36. 2026Rollout-Training Co-Design for Efficient LLM-Based Multi-Agent Reinforcement Learningcited by 2
  37. 2026BubbleSpec: Turning Long-Tail Bubbles into Speculative Rollout Drafts for Synchronous Reinforcement Learningcited by 2
  38. 2026DARTS: Distribution-Aware Active Rollout Trajectory Shaping for Accelerating LLM Reinforcement Learningcited by 2

50 most cited of 55 All 55 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Rollout-Time Intervention: Unlocking Agentic Reinforcement Learning Beyond Policy Limits
  2. 2025Slow-Fast Policy Optimization: Reposition-Before-Update for LLM Reasoning
  3. 2026Resource-Efficient Reinforcement for Reasoning Large Language Models via Dynamic One-Shot Policy Refinement
  4. 2026BASIS: Batchwise Advantage Estimation from Single-Rollout Information Sharing for LLM Reasoning
  5. 2025Training Language Models to Reason Efficiently
  6. 2025Enhancing Efficiency and Exploration in Reinforcement Learning for LLMs
  7. 2026The Art of Efficient Reasoning: Data, Reward, and Optimization
  8. 2025How Far Are We from Optimal Reasoning Efficiency?