mlsanityData to 7 Oct 2026

Improving Sampling Efficiency in RLVR through Adaptive Rollout and Response Reuse

arXiv 2509.25808 ↗First posted 30 Sep 2025cs.LGcited by 28

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Large language models (LLMs) have achieved impressive reasoning performance, with reinforcement learning with verifiable rewards (RLVR) emerging as a standard paradigm for post-training. A representative algorithm, group relative policy optimization (GRPO) (Shao et al., 2024), computes advantages by normalizing outcome rewards within response groups, but suffers from a vanishing advantage issue when all responses in a group receive identical rewards. To address this issue, we propose Adaptive Rollout and Response Reuse Policy Optimization (AR3PO), a sampling efficient RLVR algorithm that introduces two novel techniques: adaptive rollout, which dynamically allocates more responses to difficult prompts while saving computation on easier ones, and response reuse, which leverages previously generated correct responses to provide useful training signals. We compare AR3PO with strong RLVR baselines on multiple representative benchmarks using two different families of base models. Across the 7B and 8B models, AR3PO consistently outperforms GRPO and matches or surpasses DAPO (Yu et al., 2025), reducing rollout cost by up to 4.2x. On the larger 32B model, AR3PO achieves comparable performance to DAPO at similar training steps while maintaining substantially lower rollout cost.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 33 of its 37 references that are in the map. First posted on arXiv 30 Sep 2025.

Builds on

33 of 37 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025Depth-Breadth Synergy in RLVR: Unlocking LLM Reasoning Gains with Adaptive Exploration19 Aug
  2. 2025RLEP: Reinforcement Learning with Experience Replay for LLM Reasoning10 Jul
  3. 2025Improving Data Efficiency for LLM Reinforcement Fine-tuning Through Difficulty-targeted Online Data Selection and Rollout Replay5 Jun
  4. 2025Enhancing Efficiency and Exploration in Reinforcement Learning for LLMs24 May
  5. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  6. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  7. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  8. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  9. 2024RLHF Workflow: From Reward Modeling to Online RLHF13 May
  10. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  11. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 22

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  8. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  9. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  10. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  11. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  12. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
10 more
  1. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  2. 2025Group Sequence Policy Optimizationcited by 788
  3. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 556
  4. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 228
  5. 2023Nash Learning from Human Feedbackcited by 220
  6. 2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHFcited by 106
  7. 2024Online Iterative Reinforcement Learning from Human Feedback with General Preference Modelcited by 52
  8. 2024Iterative Nash Policy Optimization: Aligning LLMs with General Preferences via No-Regret Learningcited by 45
  9. 2025Improving LLM General Preference Alignment via Optimistic Online Mirror Descentcited by 24
  10. 2025Optimizing Chain-of-Thought Reasoners via Gradient Variance Minimization in Rejection Sampling and RLcited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 26.

28 papers.

Sort:

By area:

  1. 2026Self-Hinting Language Models Enhance Reinforcement Learningcited by 19
  2. 2025Stable and Efficient Single-Rollout RL for Multimodal Reasoningcited by 8
  3. 2026Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learningcited by 8
  4. 2026Prompt replay: speeding up grpo with on-policy reuse of high-signal promptscited by 7
  5. 2026RLVR without Ineffective Samples: Group Prioritized Off-Policy Optimization for LLM Reasoningcited by 7
  6. 2026Learning as Reasoning Unfolds: Progressive Rollout Allocation for Efficient Reinforcement Learningcited by 7
  7. 2026Learning from Language Feedback via Variational Policy Distillationcited by 6
  8. 2026Complementary RL: Towards Efficient Experience-Driven Agent Learningcited by 5
  9. 2026HeaPA: Difficulty-Aware Heap Sampling and On-Policy Query Augmentation for LLM Reinforcement Learningcited by 4
  10. 2026Rethinking Importance Sampling in LLM Policy Optimization: A Cumulative Token Perspectivecited by 4
  11. 2025Do Not Step Into the Same River Twice: Learning to Reason from Trial and Errorcited by 3
  12. 2026sGPO: Trading Inference FLOPs for Training Efficiency in RLVRcited by 3
16 more
  1. 2026Low-rank Optimization Trajectories Modeling for LLM RLVR Accelerationcited by 2
  2. 2026The Unlearnability Phenomenon in RLVR for Language Modelscited by 2
  3. 2026Where Rollouts Begin: Low-Load, High-Leverage First-Token Diversification for RLVRcited by 2
  4. 2026Discounted Beta-Bernoulli Reward Estimation for Sample-Efficient Reinforcement Learning with Verifiable Rewardscited by 1
  5. 2026CuSearch: Curriculum Rollout Sampling via Search Depth for Agentic RAGcited by 1
  6. 2026Learning to Solve, Forgetting to Retain: Correct-Set Turnover in RLVRcited by 1
  7. 2026Rollout-Level Advantage-Prioritized Experience Replay for GRPOcited by 1
  8. 2026A First-Principles Derivation of LLM Policy Optimization: From Expected Reward to GRPO and Its Structural Extensionscited by 1
  9. 2026SimpleSearch-VL: A Simple Recipe for Multimodal Agentic Deep Searchcited by 1
  10. 2026PAIR: Pairwise-Aware Inclusion Reweighting for Adaptive Rollout Allocation in RLVRcited by 1
  11. 2026When Do Larger Batches Help Scale LLM Reinforcement Learning?cited by 1
  12. 2026Difficulty-Adaptive Tree-Structured Policy Optimization for Expanding Reasoning Coverage in RLVRcited by 1
  13. 2026Demystifying Design Choices of Reinforcement Fine-tuning: A Batched Contextual Bandit Learning Perspectivecited by 0
  14. 2026Boosting Reinforcement Learning with Verifiable Rewards via Randomly Selected Few-Shot Guidancecited by 0
  15. 2026Tandem Reinforcement Learning with Verifiable Rewardscited by 0
  16. 2026CERO: Where and When to Allocate Rollouts for RL Post-Trainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Where to Spend Rollouts: Hit-Utility Optimal Rollout Allocation for Group-Based RLVR
  2. 2026Train Less, Learn More: Adaptive Efficient Rollout Optimization for Group-Based Reinforcement Learning
  3. 2026Revisiting Reinforcement Learning with Verifiable Rewards from a Contrastive Perspective
  4. 2026MCPO: Mastery-Consolidated Policy Optimization for Large Reasoning Models
  5. 2026Early Verdicts, Better Budgets: Sequential Adaptive Rollout Allocation for Compute-Efficient RLVR
  6. 2026Contextual Rollout Bandits for Reinforcement Learning with Verifiable Rewards
  7. 2026Beyond Negative Rollouts: Positive-Only Policy Optimization with Implicit Negative Gradients
  8. 2025Explore Data Left Behind in Reinforcement Learning for Reasoning Language Models