mlsanityData to 14 Oct 2026

Efficient RL Training for LLMs with Experience Replay

arXiv 2604.08706 ↗First posted 9 Apr 2026cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

While Experience Replay - the practice of storing rollouts and reusing them multiple times during training - is a foundational technique in general RL, it remains largely unexplored in LLM post-training due to the prevailing belief that fresh, on-policy data is essential for high performance. In this work, we challenge this assumption. We present a systematic study of replay buffers for LLM post-training, formalizing the optimal design as a trade-off between staleness-induced variance, sample diversity and the high computational cost of generation. We show that strict on-policy sampling is suboptimal when generation is expensive. Empirically, we show that a well-designed replay buffer can drastically reduce inference compute without degrading - and in some cases even improving - final model performance, while preserving policy entropy.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 25 of its 31 references that are in the map. First posted on arXiv 9 Apr 2026.

Builds on

25 of 31 references.

References · 25

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  5. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  6. 2015Continuous control with deep reinforcement learningcited by 3,407
  7. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  8. 2017Hindsight Experience Replaycited by 1,136
  9. 2015Prioritized Experience Replaycited by 1,099
  10. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 982
  11. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Modelscited by 485
  12. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 264
13 more
  1. 2024RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learningcited by 151
  2. 2017A Deeper Look at Experience Replaycited by 105
  3. 2025ARPO:End-to-End Policy Optimization for GUI Agents with Experience Replaycited by 77
  4. 2024Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Modelscited by 70
  5. 2024The Importance of Online Data: Understanding Preference Fine-tuning via Coveragecited by 55
  6. 2025Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs?cited by 53
  7. 2024Offline Regularised Reinforcement Learning for Large Language Models Alignmentcited by 48
  8. 2025LlamaRL: A Distributed Asynchronous Reinforcement Learning Framework for Efficient Large-scale LLM Trainingcited by 42
  9. 2025Trajectory Balance with Asynchrony: Decoupling Exploration and Learning for Fast, Scalable LLM Post-Trainingcited by 29
  10. 2025Asymmetric REINFORCE for off-Policy Reinforcement Learning: Balancing positive and negative rewardscited by 29
  11. 2025EFRame: Deeper Reasoning via Exploration-Filter-Replay Reinforcement Learning Frameworkcited by 14
  12. 2024Emergent properties with repeated examplescited by 11
  13. 2025RL-finetuning LLMs from on- and off-policy data with a single algorithmcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 9.

9 papers.

  1. 2026Rollout-Level Advantage-Prioritized Experience Replay for GRPOcited by 2
  2. 2026REGEN: Replay-recycling for Expert-to-Generalist distillation with Offline Reinforcement Learningcited by 2
  3. 2026Reusing Rollouts under Policy Lag: Prefix-Normalized Policy Optimization for LLM Reinforcement Learningcited by 1
  4. 2026When to Stop Reusing: Dynamic Gradient Gating for Sample-Efficient RLVRcited by 0
  5. 2026Teaching a Moving Student: Rethinking the Curriculum of On-Policy Distillationcited by 0
  6. 2026REVO: Rollout-Efficient Off-Policy Distillation via Variance-Guided Reusecited by 0
  7. 2026Self-Evolving Algorithm-Design Agents: Escaping In-Context Evolutionary Stagnation via Population-Curated Policy Optimizationcited by 0
  8. 2026Asynchronous LLM Post-Training: Group-Mass Capping and Convergence Analysiscited by 0
  9. 2026Follow the Winners: Conservative Policy Improvement with the Cross-Entropy Method for Critic-Free RFTcited by 0