mlsanityData to 14 Oct 2026

RLEP: Reinforcement Learning with Experience Replay for LLM Reasoning

arXiv 2507.07451 ↗First posted 10 Jul 2025cs.CLcited by 39

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning (RL) for large language models is an energy-intensive endeavor: training can be unstable, and the policy may gradually drift away from its pretrained weights. We present RLEP, -- ,Reinforcement Learning with Experience rePlay, -- ,a two-phase framework that first collects verified trajectories and then replays them during subsequent training. At every update step, the policy is optimized on mini-batches that blend newly generated rollouts with these replayed successes. By replaying high-quality examples, RLEP steers the model away from fruitless exploration, focuses learning on promising reasoning paths, and delivers both faster convergence and stronger final performance. On the Qwen2.5-Math-7B base model, RLEP reaches baseline peak accuracy with substantially fewer updates and ultimately surpasses it, improving accuracy on AIME-2024 from 38.2% to 39.9%, on AIME-2025 from 19.8% to 22.3%, and on AMC-2023 from 77.0% to 82.2%. Our code, datasets, and checkpoints are publicly available at https://github.com/Kwai-Klear/RLEP to facilitate reproducibility and further research.

Key facts. Cited by 39 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 13 of its 17 references that are in the map. First posted on arXiv 10 Jul 2025.

Builds on

13 of 17 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025EFRame: Deeper Reasoning via Exploration-Filter-Replay Reinforcement Learning Framework27 Jun
  2. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning2 Jun
  3. 2025Learning to Reason under Off-Policy Guidance21 Apr
  4. 2025Improving RL Exploration for LLM Reasoning through Retrospective Replay19 Apr
  5. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  6. 2025Trajectory Balance with Asynchrony: Decoupling Exploration and Learning for Fast, Scalable LLM Post-Training24 Mar
  7. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  8. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan

Other references · 5

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  4. 2015Prioritized Experience Replaycited by 1,099
  5. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 9; 2026: 30.

39 papers.

Sort:

By area:

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  2. 2025ExGRPO: Learning to Reason from Experiencecited by 53
  3. 2026HDPO: Hybrid Distillation Policy Optimization via Privileged Self-Distillationcited by 42
  4. 2025The Choice of Divergence: A Neglected Key to Mitigating Diversity Collapse in Reinforcement Learning with Verifiable Rewardcited by 37
  5. 2025Improving Sampling Efficiency in RLVR through Adaptive Rollout and Response Reusecited by 29
  6. 2025RL Grokking Recipe: How Does RL Unlock and Transfer New Algorithms in LLMs?cited by 23
  7. 2025XRPO: Pushing the limits of GRPO with Targeted Exploration and Exploitationcited by 18
  8. 2025DeepTravel: An End-to-End Agentic Reinforcement Learning Framework for Autonomous Travel Planning Agentscited by 17
  9. 2026Reuse your FLOPs: Scaling RL on Hard Problems by Conditioning on Very Off-Policy Prefixescited by 16
  10. 2025SPEC-RL: Accelerating On-Policy Reinforcement Learning with Speculative Rolloutscited by 12
  11. 2026Contextual Rollout Bandits for Reinforcement Learning with Verifiable Rewardscited by 11
  12. 2026Train at Moving Edge: Online-Verified Prompt Selection for Efficient RL Training of Large Reasoning Modelcited by 10
27 more
  1. 2026RLVR without Ineffective Samples: Group Prioritized Off-Policy Optimization for LLM Reasoningcited by 9
  2. 2026Prompt replay: speeding up grpo with on-policy reuse of high-signal promptscited by 8
  3. 2025Beyond Reasoning Gains: Mitigating General-Capability Forgetting in Large Reasoning Modelscited by 5
  4. 2026DyJR: Preserving Diversity in Reinforcement Learning with Verifiable Rewards via Dynamic Jensen-Shannon Replaycited by 4
  5. 2026Freshness-Aware Prioritized Experience Replay for LLM/VLM Reinforcement Learningcited by 3
  6. 2026Learning from the Near Future: Temporal Self-Distillation for RLVRcited by 3
  7. 2026The Unlearnability Phenomenon in RLVR for Language Modelscited by 3
  8. 2026Zone of Proximal Policy Optimization: Teacher in Prompts, Not Gradientscited by 3
  9. 2026R^3: Replay, Reflection, and Ranking Rewards for LLM Reinforcement Learningcited by 2
  10. 2026SLEA-RL: Step-Level Experience Augmented Reinforcement Learning for Multi-Turn Agentic Trainingcited by 2
  11. 2026Rollout-Level Advantage-Prioritized Experience Replay for GRPOcited by 2
  12. 2026Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learningcited by 2
  13. 2026DRIFT: Difficulty Routing Self-DIstillation with Rhythm-Gated Exploration and Success BuFfer Trainingcited by 2
  14. 2026Experience Augmented Policy Optimization for LLM Reasoningcited by 2
  15. 2026Training Large Reasoning Models Efficiently via Progressive Thought Encodingcited by 1
  16. 2026Beyond Where to Look: Trajectory-Guided Reinforcement Learning for Multimodal RLVRcited by 1
  17. 2026EasyVideoR1: Easier RL for Video Understandingcited by 1
  18. 2026Learning to Solve, Forgetting to Retain: Correct-Set Turnover in RLVRcited by 1
  19. 2026Demystifying Design Choices of Reinforcement Fine-tuning: A Batched Contextual Bandit Learning Perspectivecited by 0
  20. 2026Off-Policy Value-Based Reinforcement Learning for Large Language Modelscited by 0
  21. 2026Reasoning Depth and Environment Complexity: A Controlled Study of RLVR Data Allocation across Logical Reasoning Taskscited by 0
  22. 2026ARMOR: Stabilizing On-Policy LLM RL with Off-Policy Anchor Samplescited by 0
  23. 2026Anchoring What Matters: A Dual-Level Learning Framework for Visually-Grounded Multimodal Reasoningcited by 0
  24. 2026Towards Unbiased On-Policy Distillation for Block Diffusion Language Modelscited by 0
  25. 2026HuatuoGPT-3: RL-Only Domain Adaptation from Base Modelscited by 0
  26. 2026Measuring and Mitigating Solution Mode Collapse in RLVRcited by 0
  27. 2026Beyond Policy Rollouts: Embedding-Space Directional Exploration with Anchored Noise for LLM RLVRcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle
  2. 2026Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learning
  3. 2025RLVE: Scaling Up Reinforcement Learning for Language Models with Adaptive Verifiable Environments
  4. 2024Teaching Large Language Models to Reason with Reinforcement Learning
  5. 2025Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection
  6. 2025Stabilizing Reinforcement Learning with LLMs: Formulation and Practices
  7. 2025Accelerating RL for LLM Reasoning with Optimal Advantage Regression
  8. 2026Re2: Unlocking LLM Reasoning via Reinforcement Learning with Re-solving