# RLEP: Reinforcement Learning with Experience Replay for LLM Reasoning

arXiv: https://arxiv.org/abs/2507.07451 · HTML: https://mlsanity.com/paper/2507.07451/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 13 of its 17 references that are in the map.
- First posted on arXiv 10 Jul 2025.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)

## Builds on (most cited first, 13 of 13)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [Understanding R1-Zero-Like Training: A Critical Perspective](https://mlsanity.com/paper/2503.20783/) (2025)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement](https://mlsanity.com/paper/2409.12122/) (2024)
- [Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning](https://mlsanity.com/paper/2506.01939/) (2025)
- [Learning to Reason under Off-Policy Guidance](https://mlsanity.com/paper/2504.14945/) (2025)
- [Trajectory Balance with Asynchrony: Decoupling Exploration and Learning for Fast, Scalable LLM Post-Training](https://mlsanity.com/paper/2503.18929/) (2025)
- [Improving RL Exploration for LLM Reasoning through Retrospective Replay](https://mlsanity.com/paper/2504.14363/) (2025)
- [EFRame: Deeper Reasoning via Exploration-Filter-Replay Reinforcement Learning Framework](https://mlsanity.com/paper/2506.22200/) (2025)

## Built on by (most cited first, 30 of 39)

- [A Survey of Reinforcement Learning for Large Reasoning Models](https://mlsanity.com/paper/2509.08827/) (2025)
- [ExGRPO: Learning to Reason from Experience](https://mlsanity.com/paper/2510.02245/) (2025)
- [HDPO: Hybrid Distillation Policy Optimization via Privileged Self-Distillation](https://mlsanity.com/paper/2603.23871/) (2026)
- [The Choice of Divergence: A Neglected Key to Mitigating Diversity Collapse in Reinforcement Learning with Verifiable Reward](https://mlsanity.com/paper/2509.07430/) (2025)
- [Improving Sampling Efficiency in RLVR through Adaptive Rollout and Response Reuse](https://mlsanity.com/paper/2509.25808/) (2025)
- [RL Grokking Recipe: How Does RL Unlock and Transfer New Algorithms in LLMs?](https://mlsanity.com/paper/2509.21016/) (2025)
- [XRPO: Pushing the limits of GRPO with Targeted Exploration and Exploitation](https://mlsanity.com/paper/2510.06672/) (2025)
- [DeepTravel: An End-to-End Agentic Reinforcement Learning Framework for Autonomous Travel Planning Agents](https://mlsanity.com/paper/2509.21842/) (2025)
- [Reuse your FLOPs: Scaling RL on Hard Problems by Conditioning on Very Off-Policy Prefixes](https://mlsanity.com/paper/2601.18795/) (2026)
- [SPEC-RL: Accelerating On-Policy Reinforcement Learning with Speculative Rollouts](https://mlsanity.com/paper/2509.23232/) (2025)
- [Contextual Rollout Bandits for Reinforcement Learning with Verifiable Rewards](https://mlsanity.com/paper/2602.08499/) (2026)
- [Train at Moving Edge: Online-Verified Prompt Selection for Efficient RL Training of Large Reasoning Model](https://mlsanity.com/paper/2603.25184/) (2026)
- [RLVR without Ineffective Samples: Group Prioritized Off-Policy Optimization for LLM Reasoning](https://mlsanity.com/paper/2606.01281/) (2026)
- [Prompt replay: speeding up grpo with on-policy reuse of high-signal prompts](https://mlsanity.com/paper/2603.21177/) (2026)
- [Beyond Reasoning Gains: Mitigating General-Capability Forgetting in Large Reasoning Models](https://mlsanity.com/paper/2510.21978/) (2025)
- [DyJR: Preserving Diversity in Reinforcement Learning with Verifiable Rewards via Dynamic Jensen-Shannon Replay](https://mlsanity.com/paper/2603.16157/) (2026)
- [Freshness-Aware Prioritized Experience Replay for LLM/VLM Reinforcement Learning](https://mlsanity.com/paper/2604.16918/) (2026)
- [Learning from the Near Future: Temporal Self-Distillation for RLVR](https://mlsanity.com/paper/2604.20733/) (2026)
- [The Unlearnability Phenomenon in RLVR for Language Models](https://mlsanity.com/paper/2605.16787/) (2026)
- [Zone of Proximal Policy Optimization: Teacher in Prompts, Not Gradients](https://mlsanity.com/paper/2606.18216/) (2026)
- [R^3: Replay, Reflection, and Ranking Rewards for LLM Reinforcement Learning](https://mlsanity.com/paper/2601.19620/) (2026)
- [SLEA-RL: Step-Level Experience Augmented Reinforcement Learning for Multi-Turn Agentic Training](https://mlsanity.com/paper/2603.18079/) (2026)
- [Rollout-Level Advantage-Prioritized Experience Replay for GRPO](https://mlsanity.com/paper/2606.04560/) (2026)
- [Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learning](https://mlsanity.com/paper/2606.21943/) (2026)
- [DRIFT: Difficulty Routing Self-DIstillation with Rhythm-Gated Exploration and Success BuFfer Training](https://mlsanity.com/paper/2606.30345/) (2026)
- [Experience Augmented Policy Optimization for LLM Reasoning](https://mlsanity.com/paper/2606.30420/) (2026)
- [Training Large Reasoning Models Efficiently via Progressive Thought Encoding](https://mlsanity.com/paper/2602.16839/) (2026)
- [Beyond Where to Look: Trajectory-Guided Reinforcement Learning for Multimodal RLVR](https://mlsanity.com/paper/2603.26126/) (2026)
- [EasyVideoR1: Easier RL for Video Understanding](https://mlsanity.com/paper/2604.16893/) (2026)
- [Learning to Solve, Forgetting to Retain: Correct-Set Turnover in RLVR](https://mlsanity.com/paper/2606.03087/) (2026)

## Abstract (arXiv)

Reinforcement learning (RL) for large language models is an energy-intensive endeavor: training can be unstable, and the policy may gradually drift away from its pretrained weights. We present RLEP, -- ,Reinforcement Learning with Experience rePlay, -- ,a two-phase framework that first collects verified trajectories and then replays them during subsequent training. At every update step, the policy is optimized on mini-batches that blend newly generated rollouts with these replayed successes. By replaying high-quality examples, RLEP steers the model away from fruitless exploration, focuses learning on promising reasoning paths, and delivers both faster convergence and stronger final performance. On the Qwen2.5-Math-7B base model, RLEP reaches baseline peak accuracy with substantially fewer updates and ultimately surpasses it, improving accuracy on AIME-2024 from 38.2% to 39.9%, on AIME-2025 from 19.8% to 22.3%, and on AMC-2023 from 77.0% to 82.2%. Our code, datasets, and checkpoints are publicly available at https://github.com/Kwai-Klear/RLEP to facilitate reproducibility and further research.