mlsanityData to 14 Oct 2026

Rollout-Level Advantage-Prioritized Experience Replay for GRPO

arXiv 2606.04560 ↗First posted 3 Jun 2026cs.LG cs.AIcited by 2

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning from verifiable rewards with GRPO is a standard approach for post-training reasoning LLMs. It remains sample inefficient. Each rollout is used for a single gradient update and then discarded. Naive replay is not well suited in this setting because LLM policies drift quickly per gradient step. Stored rollouts therefore become stale and can destabilize training. We propose a rollout-level replay buffer for GRPO that stores and samples individual rollouts rather than whole groups. The buffer bounds staleness through age eviction. Any rollout older than tau_max training steps is removed. The buffer also preserves on-policy data via fresh-anchored composition. Each batch keeps its fresh on-policy rollouts and then concatenates replay rollouts drawn separately from the buffer. We prioritize replay by per-rollout advantage magnitude and recycle individual rollouts whose advantages are large. Across three Qwen3-Base scales on five math benchmarks, our method outperforms GRPO and naive replay baselines. Gains are positive at every scale and reach +1.66 pp on the five-benchmark average at 4B. Under an AES metric that jointly measures accuracy and token efficiency, our method is the only condition with a positive margin over GRPO at every scale.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 45 of its 85 references that are in the map. First posted on arXiv 3 Jun 2026.

Builds on

45 of 85 references.

Closest ancestors · 23

Same direction, newest first

  1. 2026Freshness-Aware Prioritized Experience Replay for LLM/VLM Reinforcement Learning18 Apr
  2. 2026Prompt replay: speeding up grpo with on-policy reuse of high-signal prompts22 Mar
  3. 2026DyJR: Preserving Diversity in Reinforcement Learning with Verifiable Rewards via Dynamic Jensen-Shannon Replay17 Mar
  4. 2026Unbiased Dynamic Pruning for Efficient Group-Based Policy Optimization4 Mar
  5. 2026Buffer Matters: Unleashing the Power of Off-Policy Reinforcement Learning in Large Language Model Reasoning24 Feb
  6. 2025Reinforce-Ada: An Adaptive Sampling Framework under Non-linear RL Objectives6 Oct
  7. 2025ExGRPO: Learning to Reason from Experience2 Oct
  8. 2025Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs?1 Oct
  9. 2025Improving Sampling Efficiency in RLVR through Adaptive Rollout and Response Reuse30 Sep
  10. 2025VCRL: Variance-based Curriculum Reinforcement Learning for Large Language Models24 Sep
  11. 2025Sample-efficient LLM Optimization with Reset Replay8 Aug
  12. 2025RL-PLUS: Countering Capability Boundary Collapse of LLMs in Reinforcement Learning with Hybrid-policy Optimization31 Jul
11 more
  1. 2025RLEP: Reinforcement Learning with Experience Replay for LLM Reasoning10 Jul
  2. 2025EFRame: Deeper Reasoning via Exploration-Filter-Replay Reinforcement Learning Framework27 Jun
  3. 2025RePO: Replay-Enhanced Policy Optimization11 Jun
  4. 2025SPEED-RL: Faster Training of Reasoning Models via Online Curriculum Learning10 Jun
  5. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning30 May
  6. 2025Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning18 Apr
  7. 2025A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforce15 Apr
  8. 2025DUMP: Automated Distribution-Level Curriculum Learning for RL-based LLM Post-training13 Apr
  9. 2025Efficient Reinforcement Finetuning via Adaptive Curriculum Learning7 Apr
  10. 2025Trajectory Balance with Asynchrony: Decoupling Exploration and Learning for Fast, Scalable LLM Post-Training24 Mar
  11. 2024OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework20 May

Other references · 22

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2015Continuous control with deep reinforcement learningcited by 3,407
  5. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  6. 2015Prioritized Experience Replaycited by 1,099
  7. 2025O1-Pruner: Length-Harmonizing Fine-Tuning for O1-Like Reasoning Pruningcited by 279
  8. 2018Distributed Prioritized Experience Replaycited by 269
  9. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 263
  10. 2018Deep Reinforcement Learning and the Deadly Triadcited by 174
  11. 2017A Deeper Look at Experience Replaycited by 105
  12. 2025Tapered Off-Policy REINFORCE: Stable and efficient reinforcement learning for LLMscited by 61
10 more
  1. 2025StreamRL: Scalable, Heterogeneous, and Elastic RL for LLMs with Disaggregated Stream Generationcited by 61
  2. 2025BAPO: Stabilizing Off-Policy Reinforcement Learning for LLMs via Balanced Policy Optimization with Adaptive Clippingcited by 47
  3. 2025Prompt Curriculum Learning for Efficient LLM Post-Trainingcited by 45
  4. 2025LlamaRL: A Distributed Asynchronous Reinforcement Learning Framework for Efficient Large-scale LLM Trainingcited by 42
  5. 2025AsyncFlow: An Asynchronous Streaming RL Framework for Efficient LLM Post-Trainingcited by 41
  6. 2021Reverb: A Framework For Experience Replaycited by 27
  7. 2025Optimizing Chain-of-Thought Reasoners via Gradient Variance Minimization in Rejection Sampling and RLcited by 24
  8. 2026Efficient RL Training for LLMs with Experience Replaycited by 9
  9. 2025Clip Your Sequences Fairly: Enforcing Length Fairness for Sequence-Level RLcited by 7
  10. 2026Prioritized Replay for RL Post-trainingcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Rollout Efficiency in Reinforcement Learning for Reasoning Large Language Models: A Taxonomy and Future Directionscited by 0
  2. 2026VisExMEM: Visual Evidence-Based Explainable Multimodal Evaluation Metriccited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026How Off-Policy Can GRPO Be? Mu-GRPO for Efficient LLM Reinforcement Learning
  2. 2025Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts
  3. 2026Learning as Reasoning Unfolds: Progressive Rollout Allocation for Efficient Reinforcement Learning
  4. 2026Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learning
  5. 2026GRPODropout: Less is More for Online Reinforcement Learning Rollouts
  6. 2026Learn from the Gap: Differential-Aware Advantage Pruning with Adaptive Rollout Sampling for GRPO
  7. 2026Headroom-Drift Replay: A Primitive for Principled Replay Control in GRPO
  8. 2026Train Less, Learn More: Adaptive Efficient Rollout Optimization for Group-Based Reinforcement Learning