mlsanityData to 14 Oct 2026

Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning

arXiv 2504.13818 ↗First posted 18 Apr 2025cs.LG cs.AI cs.CLcited by 84

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning with verifiable rewards (RLVR) has emerged as the leading approach for enhancing reasoning capabilities in large language models. However, it faces a fundamental compute and memory asymmetry: rollout generation is embarrassingly parallel and memory-light, whereas policy updates are communication-heavy and memory-intensive. To address this, we introduce PODS (Policy Optimization with Down-Sampling), which decouples rollout generation from policy updates by training only on a strategically selected subset of rollouts, maintaining learning quality while dramatically reducing update costs. We propose a principled subset selection criterion, max-variance down-sampling, that maximizes reward diversity, and provide an efficient O(nlog n) implementation. Empirically, Group Relative Policy Optimization (GRPO) with PODS achieves the peak test accuracy of vanilla GRPO at least 1.7× faster across the different reasoning benchmarks and hardware configurations we tested.

Key facts. Cited by 84 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 29 of its 50 references that are in the map. First posted on arXiv 18 Apr 2025.

Builds on

29 of 50 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforce15 Apr
  2. 2025VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks7 Apr
  3. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
  4. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  5. 2025What's Behind PPO's Collapse in Long-CoT? Value Optimization Holds the Secret3 Mar
  6. 2025Process Reinforcement through Implicit Rewards3 Feb
  7. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 22

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  5. 2024Qwen2.5 Technical Reportcited by 4,942
  6. 2020Learning to summarize from human feedbackcited by 2,588
  7. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  8. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  9. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  10. 2015Prioritized Experience Replaycited by 1,099
  11. 2023DataComp: In search of the next generation of multimodal datasetscited by 644
  12. 2024Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learningcited by 241
10 more
  1. 2025An Empirical Study on Eliciting and Improving R1-like Reasoning Modelscited by 78
  2. 2024Scaling Laws for Data Filtering -- Data Curation cannot be Compute Agnosticcited by 77
  3. 2025What Makes a Reward Model a Good Teacher? An Optimization Perspectivecited by 73
  4. 2025Robust Autonomy Emerges from Self-Playcited by 67
  5. 2019Learning From Less Data: A Unified Data Subset Selection and Active Learning Framework for Computer Visioncited by 64
  6. 2024Interpretable Contrastive Monte Carlo Tree Search Reasoningcited by 64
  7. 2022Actor Prioritized Experience Replaycited by 20
  8. 2023REDUCR: Robust Data Downsampling Using Class Priority Reweightingcited by 3
  9. 2022Bidder Subset Selection Problem in Auction Designcited by 1
  10. 2023Bidder Selection Problem in Position Auctions: A Fast and Simple Algorithm via Poisson Approximationcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 35; 2026: 49.

84 papers.

Sort:

By area:

  1. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoningcited by 162
  2. 2025Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rolloutscited by 113
  3. 2025Geometric-Mean Policy Optimizationcited by 111
  4. 2025SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy Optimizationcited by 89
  5. 2025BREAD: Branched Rollouts from Expert Anchors Bridge SFT & RL for Reasoningcited by 63
  6. 2025Prompt Curriculum Learning for Efficient LLM Post-Trainingcited by 45
  7. 2025RePO: Replay-Enhanced Policy Optimizationcited by 39
  8. 2025VCRL: Variance-based Curriculum Reinforcement Learning for Large Language Modelscited by 36
  9. 2025SPEED-RL: Faster Training of Reasoning Models via Online Curriculum Learningcited by 29
  10. 2025Beyond Correctness: Harmonizing Process and Outcome Rewards through RL Trainingcited by 29
  11. 2025Leanabell-Prover-V2: Verifier-integrated Reasoning for Formal Theorem Proving via Reinforcement Learningcited by 27
  12. 2025Reinforce-Ada: An Adaptive Sampling Framework under Non-linear RL Objectivescited by 25
38 more
  1. 2025TinyV: Reducing False Negatives in Verification Improves RL for LLM Reasoningcited by 21
  2. 2026Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statisticcited by 21
  3. 2025CurES: From Gradient Analysis to Efficient Curriculum Learning for Reasoning LLMscited by 19
  4. 2026Adaptive Rollout Allocation for Online Reinforcement Learning with Verifiable Rewardscited by 17
  5. 2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewardscited by 14
  6. 2025Don't Tell the Answer, Truly Guide the Reasoning During RL Rolloutscited by 14
  7. 2026Small Generalizable Prompt Predictive Models Can Steer Efficient RL Post-Training of Large Reasoning Modelscited by 14
  8. 2025Lookahead Tree-Based Rollouts for Enhanced Trajectory-Level Exploration in Reinforcement Learning with Verifiable Rewardscited by 13
  9. 2025MiroMind-M1: An Open-Source Advancement in Mathematical Reasoning via Context-Aware Multi-Stage Policy Optimizationcited by 12
  10. 2025Position: The Hidden Costs and Measurement Gaps of Reinforcement Learning with Verifiable Rewardscited by 12
  11. 2025SPEC-RL: Accelerating On-Policy Reinforcement Learning with Speculative Rolloutscited by 12
  12. 2026Learning to Hint for Reinforcement Learningcited by 12
  13. 2025A Snapshot of Influence: A Local Data Attribution Framework for Online Reinforcement Learningcited by 11
  14. 2025Unlocking the Power of Multi-Agent LLM for Reasoning: From Lazy Agents to Deliberationcited by 11
  15. 2026Contextual Rollout Bandits for Reinforcement Learning with Verifiable Rewardscited by 11
  16. 2025Let it Calm: Exploratory Annealed Decoding for Verifiable Reinforcement Learningcited by 10
  17. 2026Prune as You Generate: Online Rollout Pruning for Faster and Better RLVRcited by 10
  18. 2025LSPO: Length-aware Dynamic Sampling for Policy Optimization in LLM Reasoningcited by 9
  19. 2026Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learningcited by 9
  20. 2025Exploring Multi-Temperature Strategies for Token- and Rollout-Level Control in RLVRcited by 8
  21. 2025Stable and Efficient Single-Rollout RL for Multimodal Reasoningcited by 8
  22. 2026QuRL: Efficient Reinforcement Learning with Quantized Rolloutcited by 8
  23. 2026Orchard: An Open-Source Agentic Modeling Frameworkcited by 8
  24. 2025Efficient Reinforcement Learning for Large Language Models with Intrinsic Explorationcited by 7
  25. 2025Explore Data Left Behind in Reinforcement Learning for Reasoning Language Modelscited by 7
  26. 2026Learning as Reasoning Unfolds: Progressive Rollout Allocation for Efficient Reinforcement Learningcited by 7
  27. 2025MURPHY: Feedback-Aware GRPO with Retrospective Credit Assignment for Multi-Turn Code Generationcited by 5
  28. 2026Spend Your Rollouts Where It Counts: Rollout Allocation for Group-Based RL Post-Trainingcited by 5
  29. 2025Composing Policy Gradients and Prompt Optimization for Language Model Programscited by 4
  30. 2026HeaPA: Difficulty-Aware Heap Sampling and On-Policy Query Augmentation for LLM Reinforcement Learningcited by 4
  31. 2026Bridging Online and Offline RL: Contextual Bandit Learning for Multi-Turn Code Generationcited by 4
  32. 2025ToolSample: Dual Dynamic Sampling Methods with Curriculum Learning for RL-based Tool Learningcited by 3
  33. 2026Resource-Efficient Reinforcement for Reasoning Large Language Models via Dynamic One-Shot Policy Refinementcited by 3
  34. 2026What Does Preference Learning Recover from Pairwise Comparison Data?cited by 3
  35. 2026Not all tokens are needed(NAT): token efficient reinforcement learningcited by 3
  36. 2026Supervised Fine-Tuning versus Reinforcement Learning: A Study of Post-Training Methods for Large Language Modelscited by 3
  37. 2026CoDistill-GRPO: A Co-Distillation Recipe for Efficient Group Relative Policy Optimizationcited by 3
  38. 2026The Unlearnability Phenomenon in RLVR for Language Modelscited by 3

50 most cited of 84 All 84 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026GRPODropout: Less is More for Online Reinforcement Learning Rollouts
  2. 2026Reusing Rollouts under Policy Lag: Prefix-Normalized Policy Optimization for LLM Reinforcement Learning
  3. 2026Double-Rollout Reinforcement Learning with Sampling-based Reward Estimation
  4. 2026Where to Spend Rollouts: Hit-Utility Optimal Rollout Allocation for Group-Based RLVR
  5. 2026Train Less, Learn More: Adaptive Efficient Rollout Optimization for Group-Based Reinforcement Learning
  6. 2026Beyond Negative Rollouts: Positive-Only Policy Optimization with Implicit Negative Gradients
  7. 2026RolloutPipe: Overlapping Pipelined Rollout and Training in Disaggregated On-Policy LLM Reinforcement Learning
  8. 2026R2PO: Decoupling Rollout and Inference Policies for LLM Reasoning