# Contextual Rollout Bandits for Reinforcement Learning with Verifiable Rewards

arXiv: https://arxiv.org/abs/2602.08499 · HTML: https://mlsanity.com/paper/2602.08499/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 51 of its 70 references that are in the map.
- First posted on arXiv 9 Feb 2026.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)

## Builds on (most cited first, 30 of 51)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [Solving Quantitative Reasoning Problems with Language Models](https://mlsanity.com/paper/2206.14858/) (2022)
- [Understanding R1-Zero-Like Training: A Critical Perspective](https://mlsanity.com/paper/2503.20783/) (2025)
- [s1: Simple test-time scaling](https://mlsanity.com/paper/2501.19393/) (2025)
- [OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems](https://mlsanity.com/paper/2402.14008/) (2024)
- [Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?](https://mlsanity.com/paper/2504.13837/) (2025)
- [A General Theoretical Paradigm to Understand Learning from Human Preferences](https://mlsanity.com/paper/2310.12036/) (2023)
- [Group Sequence Policy Optimization](https://mlsanity.com/paper/2507.18071/) (2025)
- [Safe RLHF: Safe Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2310.12773/) (2023)
- [LESS: Selecting Influential Data for Targeted Instruction Tuning](https://mlsanity.com/paper/2402.04333/) (2024)
- [Chain-of-Thought Reasoning In The Wild Is Not Always Faithful](https://mlsanity.com/paper/2503.08679/) (2025)
- [The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning](https://mlsanity.com/paper/2506.01347/) (2025)
- [Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts](https://mlsanity.com/paper/2506.02177/) (2025)
- [#InsTag: Instruction Tagging for Analyzing Supervised Fine-tuning of Large Language Models](https://mlsanity.com/paper/2308.07074/) (2023)
- [Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning](https://mlsanity.com/paper/2504.13818/) (2025)
- [Small ReLU networks are powerful memorizers: a tight analysis of memorization capacity](https://mlsanity.com/paper/1810.07770/) (2018)
- [Active Preference Optimization for Sample Efficient RLHF](https://mlsanity.com/paper/2402.10500/) (2024)
- [Maybe Only 0.5% Data is Needed: A Preliminary Exploration of Low Training Data Instruction Tuning](https://mlsanity.com/paper/2305.09246/) (2023)
- [ExGRPO: Learning to Reason from Experience](https://mlsanity.com/paper/2510.02245/) (2025)
- [Re-Mix: Optimizing Data Mixtures for Large Scale Imitation Learning](https://mlsanity.com/paper/2408.14037/) (2024)
- [EE-Net: Exploitation-Exploration Neural Networks in Contextual Bandits](https://mlsanity.com/paper/2110.03177/) (2021)
- [DCPO: Dynamic Clipping Policy Optimization](https://mlsanity.com/paper/2509.02333/) (2025)

## Built on by (most cited first, 11 of 11)

- [Counterfactual Credit Policy Optimization for Multi-Agent Collaboration](https://mlsanity.com/paper/2603.21563/) (2026)
- [When Self-Belief Misleads: Active Label Acquisition for Reinforcement Learning with Verifiable Rewards](https://mlsanity.com/paper/2605.25864/) (2026)
- [World Models: A Comprehensive Survey of Architectures, Methodologies, Reasoning Paradigms, and Applications](https://mlsanity.com/paper/2606.00133/) (2026)
- [Adaptive Robust Estimator for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2603.21574/) (2026)
- [Policy Improvement Reinforcement Learning](https://mlsanity.com/paper/2604.00860/) (2026)
- [AMR-SD: Asymmetric Meta-Reflective Self-Distillation for Token-Level Credit Assignment](https://mlsanity.com/paper/2605.18529/) (2026)
- [Implicit Hierarchical GRPO: Decoupling Tool Invocation from Execution for Tool-Integrated Mathematical Reasoning](https://mlsanity.com/paper/2605.18500/) (2026)
- [Toward Plasticity-Preserving KL Regularization for Capability Retention in LLM Reinforcement Learning](https://mlsanity.com/paper/2608.01743/) (2026)
- [When Not to Imitate: Boundary-Aware Skill Memory for Reliable Tool-Use LLM Agents](https://mlsanity.com/paper/2608.22339/) (2026)
- [TGRL: Temperature-Grouped Reinforcement Learning for Efficient Exploration in LLMs](https://mlsanity.com/paper/2609.33589/) (2026)
- [CARM: Cancellation-Aware Response Masking for LLM Reinforcement Learning](https://mlsanity.com/paper/2610.02039/) (2026)

## Abstract (arXiv)

Reinforcement Learning with Verifiable Rewards (RLVR) is an effective paradigm for improving the reasoning capabilities of large language models. However, existing RLVR methods utilize rollouts in an indiscriminate and short-horizon manner: responses of heterogeneous quality within each prompt are treated uniformly, and historical rollouts are discarded after a single use. This leads to noisy supervision, poor sample efficiency, and suboptimal policy updates. We address these issues by formulating rollout scheduling in RLVR as a contextual bandit problem and proposing a unified neural scheduling framework that adaptively selects high-value rollouts throughout training. Each rollout is treated as an arm whose reward is defined by the induced performance gain between consecutive optimization steps. The resulting scheduler supports both noise-aware intra-group selection and adaptive global reuse of historical rollouts within a single principled framework. We provide theoretical justification by deriving sublinear regret bounds and showing that enlarging the rollout buffer improves the achievable performance upper bound. Experiments on six mathematical reasoning benchmarks demonstrate consistent gains in performance and training efficiency across multiple RLVR optimization methods.