mlsanityData to 14 Oct 2026

Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts

arXiv 2506.02177 ↗First posted 2 Jun 2025cs.AI cs.LGcited by 113

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning, such as PPO and GRPO, has powered recent breakthroughs in LLM reasoning. Scaling rollout to sample more prompts enables models to selectively use higher-quality data for training, which can stabilize RL training and improve model performance. However, this comes at the cost of significant computational overhead. In this paper, we show that a substantial portion of this overhead can be avoided by skipping uninformative prompts before rollout. Our analysis of reward dynamics reveals a strong temporal consistency in prompt value: prompts that are uninformative in one epoch of training are likely to remain uninformative in future epochs. Based on these insights, we propose GRESO (GRPO with Efficient Selective Rollout), an online, lightweight pre-rollout filtering algorithm that predicts and skips uninformative prompts using reward training dynamics. By evaluating GRESO on a broad range of math reasoning benchmarks and models, such as Qwen2.5-Math-1.5B, DeepSeek-R1-Distill-Qwen-1.5B, and Qwen2.5-Math-7B, we show that GRESO achieves up to 2.4x wall-clock time speedup in rollout and up to 2.0x speedup in total training time without accuracy degradation.

Key facts. Cited by 113 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 43 of its 57 references that are in the map. First posted on arXiv 2 Jun 2025.

Builds on

43 of 57 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Example29 Apr
  2. 2025Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning18 Apr
  3. 2025A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforce15 Apr
  4. 2025VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks7 Apr
  5. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  6. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  7. 2025What's Behind PPO's Collapse in Long-CoT? Value Optimization Holds the Secret3 Mar
  8. 2025LIMR: Less is More for RL Scaling17 Feb
  9. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  10. 2024On Designing Effective RL Reward at Training Time for LLM Reasoning19 Oct
  11. 2023Let's Verify Step by Step31 May

Other references · 32

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  5. 2017Deep reinforcement learning from human preferencescited by 3,912
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  7. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  8. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  9. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  10. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  11. 2025s1: Simple test-time scalingcited by 1,341
  12. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
20 more
  1. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  2. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 678
  3. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  4. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  5. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 532
  6. 2025LIMO: Less is More for Reasoningcited by 498
  7. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervisioncited by 318
  8. 2023Evaluating the Performance of Large Language Models on GAOKAO Benchmarkcited by 186
  9. 2022Coverage-centric Coreset Selection for High Pruning Ratescited by 117
  10. 2025Concise Reasoning via Reinforcement Learningcited by 79
  11. 2024Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Modelscited by 70
  12. 2024Active Preference Optimization for Sample Efficient RLHFcited by 62
  13. 2024Active Preference Learning for Large Language Modelscited by 58
  14. 2023SALMON: Self-Alignment with Instructable Reward Modelscited by 55
  15. 2024Optimizing RLHF Training for Large Language Models with Stage Fusioncited by 43
  16. 2025Large-Scale Data Selection for Instruction Tuningcited by 35
  17. 2022Data-Efficient Finetuning Using Cross-Task Nearest Neighborscited by 26
  18. 2023Make Every Example Count: On the Stability and Utility of Self-Influence for Learning from Noisy NLP Datasetscited by 19
  19. 2024ELFS: Label-Free Coreset Selection with Proxy Training Dynamicscited by 10
  20. 2024Enabling Weak LLMs to Judge Response Reliability via Meta Rankingcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 28; 2026: 85.

113 papers.

Sort:

By area:

  1. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
  2. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  3. 2025The Art of Scaling Reinforcement Learning Compute for LLMscited by 101
  4. 2025Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs?cited by 53
  5. 2025Can Prompt Difficulty be Online Predicted for Accelerating RL Finetuning of Reasoning Models?cited by 52
  6. 2025No Prompt Left Behind: Exploiting Zero-Variance Prompts in LLM Reinforcement Learning via Entropy-Guided Advantage Shapingcited by 50
  7. 2025Prompt Curriculum Learning for Efficient LLM Post-Trainingcited by 45
  8. 2025RollPacker: Mitigating Long-Tail Rollouts for Fast, Synchronous RL Post-Trainingcited by 35
  9. 2025Single-stream Policy Optimizationcited by 28
  10. 2025It Takes Two: Your GRPO Is Secretly DPOcited by 26
  11. 2025Reinforce-Ada: An Adaptive Sampling Framework under Non-linear RL Objectivescited by 25
  12. 2025Towards High Data Efficiency in Reinforcement Learning with Verifiable Rewardcited by 19
38 more
  1. 2026Does Your Reasoning Model Implicitly Know When to Stop Thinking?cited by 19
  2. 2025XRPO: Pushing the limits of GRPO with Targeted Exploration and Exploitationcited by 18
  3. 2026Adaptive Rollout Allocation for Online Reinforcement Learning with Verifiable Rewardscited by 17
  4. 2026Devil is in Narrow Policy: Unleashing Exploration in Driving VLA Modelscited by 15
  5. 2026Small Generalizable Prompt Predictive Models Can Steer Efficient RL Post-Training of Large Reasoning Modelscited by 14
  6. 2025Position: The Hidden Costs and Measurement Gaps of Reinforcement Learning with Verifiable Rewardscited by 12
  7. 2025SPEC-RL: Accelerating On-Policy Reinforcement Learning with Speculative Rolloutscited by 12
  8. 2025RLBoost: Harvesting Preemptible Resources for Cost-Efficient Reinforcement Learning on LLMscited by 11
  9. 2026Contextual Rollout Bandits for Reinforcement Learning with Verifiable Rewardscited by 11
  10. 2026Dynamics-Predictive Sampling for Active RL Finetuning of Large Reasoning Modelscited by 11
  11. 2026Agent psychometrics: Task-level performance prediction in agentic coding benchmarkscited by 11
  12. 2026Escaping the KL Agreement Trap in On-Policy Distillationcited by 11
  13. 2026Distill Where You Fail: Recovering Learning Signals of Negative RL-Groups from Adaptive Teacher Guidancecited by 11
  14. 2025NaviMaster: Learning a Unified Policy for GUI and Embodied Navigation Taskscited by 10
  15. 2025G2RPO-A: Guided Group Relative Policy Optimization with Adaptive Guidancecited by 10
  16. 2025Let it Calm: Exploratory Annealed Decoding for Verifiable Reinforcement Learningcited by 10
  17. 2026Prune as You Generate: Online Rollout Pruning for Faster and Better RLVRcited by 10
  18. 2026TRACE: A Unified Rollout Budget Allocation Framework for Efficient Agentic Reinforcement Learningcited by 10
  19. 2026Understanding Diversity Collapse in RLVR via the Lens of Overtrainingcited by 10
  20. 2025FastGRPO: Accelerating Policy Optimization via Concurrency-aware Speculative Decoding and Online Draft Learningcited by 9
  21. 2025LSPO: Length-aware Dynamic Sampling for Policy Optimization in LLM Reasoningcited by 9
  22. 2026Accelerating RL Post-Training Rollouts via System-Integrated Speculative Decodingcited by 9
  23. 2026Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learningcited by 9
  24. 2026RLVR without Ineffective Samples: Group Prioritized Off-Policy Optimization for LLM Reasoningcited by 9
  25. 2025Stable and Efficient Single-Rollout RL for Multimodal Reasoningcited by 8
  26. 2026QuRL: Efficient Reinforcement Learning with Quantized Rolloutcited by 8
  27. 2026Actor-Curator: Co-adaptive Curriculum Learning via Policy-Improvement Bandits for RL Post-Trainingcited by 8
  28. 2026Prompt replay: speeding up grpo with on-policy reuse of high-signal promptscited by 8
  29. 2026Orchard: An Open-Source Agentic Modeling Frameworkcited by 8
  30. 2025Efficient Reinforcement Learning for Large Language Models with Intrinsic Explorationcited by 7
  31. 2025Explore Data Left Behind in Reinforcement Learning for Reasoning Language Modelscited by 7
  32. 2026V0: A Generalist Value Model for Any Policy at State Zerocited by 7
  33. 2026Learning from Language Feedback via Variational Policy Distillationcited by 7
  34. 2026Autonomous Continual Learning for Environment Adaptation of Computer-Use Agentscited by 6
  35. 2026Composition-RL: Compose Your Verifiable Prompts for Reinforcement Learning of Large Language Modelscited by 6
  36. 2026SRT: Accelerating Reinforcement Learning via Speculative Rollout with Tree-Structured Cachecited by 5
  37. 2026Jackpot: Optimal Budgeted Rejection Sampling for Extreme Actor-Policy Mismatch Reinforcement Learningcited by 5
  38. 2026Your Language Model is Its Own Critic: Reinforcement Learning with Value Estimation from Actor's Internal Statescited by 5

50 most cited of 113 All 113 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Learning as Reasoning Unfolds: Progressive Rollout Allocation for Efficient Reinforcement Learning
  2. 2026BASIS: Batchwise Advantage Estimation from Single-Rollout Information Sharing for LLM Reasoning
  3. 2026R2PO: Decoupling Rollout and Inference Policies for LLM Reasoning
  4. 2026Train at Moving Edge: Online-Verified Prompt Selection for Efficient RL Training of Large Reasoning Model
  5. 2025Can GRPO Help LLMs Transcend Their Pretraining Origin?
  6. 2026Rollout-Level Advantage-Prioritized Experience Replay for GRPO
  7. 2026Rollout Efficiency in Reinforcement Learning for Reasoning Large Language Models: A Taxonomy and Future Directions
  8. 2025Accelerating RL for LLM Reasoning with Optimal Advantage Regression