# CERO: Where and When to Allocate Rollouts for RL Post-Training

arXiv: https://arxiv.org/abs/2610.09679 · HTML: https://mlsanity.com/paper/2610.09679/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 19 of its 39 references that are in the map.
- First posted on arXiv 7 Oct 2026.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 19 of 19)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [Self-Evolving Curriculum for LLM Reasoning](https://mlsanity.com/paper/2505.14970/) (2025)
- [CPPO: Accelerating the Training of Group Relative Policy Optimization-Based Reasoning Models](https://mlsanity.com/paper/2503.22342/) (2025)
- [Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning](https://mlsanity.com/paper/2504.13818/) (2025)
- [Sample More to Think Less: Group Filtered Policy Optimization for Concise Reasoning](https://mlsanity.com/paper/2508.09726/) (2025)
- [SPEED-RL: Faster Training of Reasoning Models via Online Curriculum Learning](https://mlsanity.com/paper/2506.09016/) (2025)
- [Improving Sampling Efficiency in RLVR through Adaptive Rollout and Response Reuse](https://mlsanity.com/paper/2509.25808/) (2025)
- [Knapsack RL: Unlocking Exploration of LLMs via Optimizing Budget Allocation](https://mlsanity.com/paper/2509.25849/) (2025)
- [Train Less, Learn More: Adaptive Efficient Rollout Optimization for Group-Based Reinforcement Learning](https://mlsanity.com/paper/2602.14338/) (2026)
- [CoBA-RL: Capability-Oriented Budget Allocation for Reinforcement Learning in LLMs](https://mlsanity.com/paper/2602.03048/) (2026)
- [VADE: Variance-Aware Dynamic Sampling via Online Sample-Level Difficulty Estimation for Multimodal RL](https://mlsanity.com/paper/2511.18902/) (2025)
- [Learning as Reasoning Unfolds: Progressive Rollout Allocation for Efficient Reinforcement Learning](https://mlsanity.com/paper/2607.22002/) (2026)
- [Where to Spend Rollouts: Hit-Utility Optimal Rollout Allocation for Group-Based RLVR](https://mlsanity.com/paper/2605.07114/) (2026)
- [Spend Your Rollouts Where It Counts: Rollout Allocation for Group-Based RL Post-Training](https://mlsanity.com/paper/2605.26606/) (2026)
- [Goldilocks RL: Tuning Task Difficulty to Escape Sparse Rewards for Reasoning](https://mlsanity.com/paper/2602.14868/) (2026)

## Abstract (arXiv)

Adaptive rollout methods for group-relative reinforcement learning typically allocate a fixed per-update budget across prompts. We instead study how to coordinate a finite rollout budget over the entire training horizon. We formulate this problem using a concave surrogate utility of cumulative prompt exposure and introduce CERO, an online primal dual scheduler for prompt admission and budget pacing. In our experiments, each admitted prompt receives a fixed-size response group. CERO instead adapts which prompts are selected, how often they are revisited across rounds, and how many groups are generated in each round. A compact Fenchel representation linearizes the dependence on cumulative exposure, while projected online gradient descent updates prompt-specific supporting slopes and a shared budget price using reward-variation feedback and budget deviations. We establish pathwise guarantees for the surrogate allocation objective against fixed-rate and same-path time-varying benchmarks, with explicit terms for proxy discrepancy and rate variation. Under matched training-response budgets, CERO attains the highest avg@16 macro-average on each of three backbones across five mathematical reasoning benchmarks. Mechanistic analyses link CERO's prompt choices to within-group reward contrast, while multi-seed ablations show gains from adaptive pacing over both uniform and preset spending schedules.