mlsanityData to 7 Oct 2026

Early Verdicts, Better Budgets: Sequential Adaptive Rollout Allocation for Compute-Efficient RLVR

arXiv 2607.26253 ↗First posted 28 Jul 2026cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

Reinforcement learning with verifiable rewards (RLVR) is bottlenecked by rollout generation, yet many sampled prompts produce saturated groups (all responses correct or all incorrect) whose zero reward variance yields no policy-gradient signal. Existing remedies either oversample a larger candidate pool and discard saturated prompts (dynamic sampling), paying heavy extra rollouts, or predict prompt difficulty before sampling, which is fragile under a shifting policy. We observe that a group's effectiveness is usually decided early, within the first few of its rollouts, so spending a full group on an already-decided prompt is wasteful. We cast per-step rollout collection as a budget-constrained sequential allocation (optimal stopping) problem and introduce SARA (Sequential Adaptive Rollout Allocation). SARA maintains a Beta posterior over each prompt's success rate, evaluates a closed-form predictor of group effectiveness, and applies a two-threshold, SPRT-style rule that commits effective groups, abandons saturated ones after a short probe, and reallocates the freed budget to fresh prompts, without any extra prediction rollouts. We prove abandonment reliability, expected rollout savings, fixed-budget yield dominance, and a link between effective-group yield and the GRPO gradient norm. On mathematical reasoning and planning with 1.5B/3B models on a single GPU, SARA matches DPS (both below the DS oracle) while using 22% fewer rollouts than DS; composing SARA with DPS yields the best accuracy, slightly above DS, at 67% fewer rollouts (near-uniform cost).

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 30 of its 38 references that are in the map. First posted on arXiv 28 Jul 2026.

Builds on

30 of 38 references.

References · 30

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  4. 2024Qwen2.5 Technical Reportcited by 4,713
  5. 2023Let's Verify Step by Stepcited by 3,598
  6. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  7. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  8. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  9. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
  10. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
  11. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  12. 2025LIMO: Less is More for Reasoningcited by 488
18 more
  1. 2025Stop Overthinking: A Survey on Efficient Reasoning for Large Language Modelscited by 456
  2. 2025L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learningcited by 381
  3. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  4. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Examplecited by 288
  5. 2013Learning to Optimize Via Posterior Samplingcited by 253
  6. 2025REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalizationcited by 233
  7. 2025ThinkPrune: Pruning Long Chain-of-Thought of LLMs via Reinforcement Learningcited by 187
  8. 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Modelscited by 187
  9. 2025LIMR: Less is More for RL Scalingcited by 136
  10. 2025Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rolloutscited by 108
  11. 2025Self-Evolving Curriculum for LLM Reasoningcited by 107
  12. 2025Online Difficulty Filtering for Reasoning Oriented Reinforcement Learningcited by 100
  13. 2025CPPO: Accelerating the Training of Group Relative Policy Optimization-Based Reasoning Modelscited by 93
  14. 2025Concise Reasoning via Reinforcement Learningcited by 78
  15. 2025SRPO: A Cross-Domain Implementation of Large-Scale Reinforcement Learning on LLMcited by 67
  16. 2026Small Generalizable Prompt Predictive Models Can Steer Efficient RL Post-Training of Large Reasoning Modelscited by 11
  17. 2026TRACE: A Unified Rollout Budget Allocation Framework for Efficient Agentic Reinforcement Learningcited by 9
  18. 2026Dynamics-Predictive Sampling for Active RL Finetuning of Large Reasoning Modelscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026SIPO: Selective-Inference Policy Optimization for Tree-Structured Agentic RLcited by 0
  2. 2026SERA: Scale-Equalized Rollout Allocation for Maximum Likelihood Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Where to Spend Rollouts: Hit-Utility Optimal Rollout Allocation for Group-Based RLVR
  2. 2026CERO: Where and When to Allocate Rollouts for RL Post-Training
  3. 2026PAIR: Pairwise-Aware Inclusion Reweighting for Adaptive Rollout Allocation in RLVR
  4. 2026Contextual Rollout Bandits for Reinforcement Learning with Verifiable Rewards
  5. 2026Adaptive Rollout Allocation for Online Reinforcement Learning with Verifiable Rewards
  6. 2026Spend Your Rollouts Where It Counts: Rollout Allocation for Group-Based RL Post-Training
  7. 2025Improving Sampling Efficiency in RLVR through Adaptive Rollout and Response Reuse
  8. 2026Where Rollouts Begin: Low-Load, High-Leverage First-Token Diversification for RLVR