mlsanityData to 7 Oct 2026

RePO: Replay-Enhanced Policy Optimization

arXiv 2506.09340 ↗First posted 11 Jun 2025cs.CL cs.AI cs.LGcited by 36

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning (RL) is vital for optimizing large language models (LLMs). Recent Group Relative Policy Optimization (GRPO) estimates advantages using multiple on-policy outputs per prompt, leading to high computational costs and low data efficiency. To address this, we introduce Replay-Enhanced Policy Optimization (RePO), which leverages diverse replay strategies to retrieve off-policy samples from a replay buffer, allowing policy optimization based on a broader and more diverse set of samples for each prompt. Experiments on five LLMs across seven mathematical reasoning benchmarks demonstrate that RePO achieves absolute average performance gains of 18.4 and 4.1 points for Qwen2.5-Math-1.5B and Qwen3-1.7B, respectively, compared to GRPO. Further analysis indicates that RePO increases computational cost by 15% while raising the number of effective optimization steps by 48% for Qwen3-1.7B, with both on-policy and off-policy sample numbers set to 8. The repository can be accessed at https://github.com/SihengLi99/RePO.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 24 of its 32 references that are in the map. First posted on arXiv 11 Jun 2025.

Builds on

24 of 32 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025Learning to Reason under Off-Policy Guidance21 Apr
  2. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  3. 2025Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning18 Apr
  4. 2025DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning15 Apr
  5. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
  6. 2025CPPO: Accelerating the Training of Group Relative Policy Optimization-Based Reasoning Models28 Mar
  7. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  8. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  9. 2024Teaching Large Language Models to Reason with Reinforcement Learning7 Mar
  10. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 14

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  5. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  6. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  7. 2020Learning to summarize from human feedbackcited by 2,562
  8. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  9. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  10. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  11. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  12. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
2 more
  1. 2025What Makes a Reward Model a Good Teacher? An Optimization Perspectivecited by 70
  2. 2023Vanishing Gradients in Reinforcement Finetuning of Language Modelscited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 11; 2026: 25.

36 papers.

Sort:

By area:

  1. 2025Geometric-Mean Policy Optimizationcited by 107
  2. 2025ExGRPO: Learning to Reason from Experiencecited by 49
  3. 2025MO-GRPO: Mitigating Reward Hacking of Group Relative Policy Optimization on Multi-Objective Problemscited by 21
  4. 2026Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statisticcited by 19
  5. 2025XRPO: Pushing the limits of GRPO with Targeted Exploration and Exploitationcited by 17
  6. 2025Squeeze the Soaked Sponge: Efficient Off-policy Reinforcement Finetuning for Large Language Modelcited by 14
  7. 2025AutoTool: Dynamic Tool Selection and Integration for Agentic Reasoningcited by 11
  8. 2026Good SFT Optimizes for SFT, Better SFT Prepares for Reinforcement Learningcited by 10
  9. 2026Train at Moving Edge: Online-Verified Prompt Selection for Efficient RL Training of Large Reasoning Modelcited by 10
  10. 2025AVATAR: Reinforcement Learning to See, Hear, and Reason Over Videocited by 9
  11. 2025Stabilizing Off-Policy Training for Long-Horizon LLM Agent via Turn-Level Importance Sampling and Clipping-Triggered Normalizationcited by 8
  12. 2026Buffer Matters: Unleashing the Power of Off-Policy Reinforcement Learning in Large Language Model Reasoningcited by 8
24 more
  1. 2026Accelerating RL Post-Training Rollouts via System-Integrated Speculative Decodingcited by 8
  2. 2026OP-GRPO: Efficient Off-Policy GRPO for Flow-Matching Modelscited by 7
  3. 2026RLVR without Ineffective Samples: Group Prioritized Off-Policy Optimization for LLM Reasoningcited by 7
  4. 2026Large Language Model Post-Training: A Unified View of Off-Policy and On-Policy Learningcited by 5
  5. 2025PVPO: Pre-Estimated Value-Based Policy Optimization for Agentic Reasoningcited by 4
  6. 2026DyJR: Preserving Diversity in Reinforcement Learning with Verifiable Rewards via Dynamic Jensen-Shannon Replaycited by 3
  7. 2026Learning from the Near Future: Temporal Self-Distillation for RLVRcited by 3
  8. 2026Kernelized Advantage Estimation: From Nonparametric Statistics to LLM Reasoningcited by 3
  9. 2025From Poisoned to Aware: Fostering Backdoor Self-Awareness in LLMscited by 2
  10. 2026Experience Augmented Policy Optimization for LLM Reasoningcited by 2
  11. 2026One Ring to Rule Them All: Unifying Group-Based RL via Dynamic Power-Mean Geometrycited by 1
  12. 2026Decoding in Geometry: Alleviating Embedding-Space Crowding for Complex Reasoningcited by 1
  13. 2026PrAg-PO: Prompt Augmented Policy Optimization for Robust and Diverse Mathematical Reasoningcited by 1
  14. 2026Learning to Solve, Forgetting to Retain: Correct-Set Turnover in RLVRcited by 1
  15. 2026Rollout-Level Advantage-Prioritized Experience Replay for GRPOcited by 1
  16. 2026A First-Principles Derivation of LLM Policy Optimization: From Expected Reward to GRPO and Its Structural Extensionscited by 1
  17. 2026Headroom-Drift Replay: A Primitive for Principled Replay Control in GRPOcited by 1
  18. 2025Scaling DRL for Decision Making: A Survey on Data, Network, and Training Budget Strategiescited by 0
  19. 2026Beyond Normalization: Rethinking the Partition Function as a Difficulty Scheduler for RLVRcited by 0
  20. 2026Chart Deep Research in LVLMs via Parallel Relative Policy Optimizationcited by 0
  21. 2026RSPO: Reward-Swap Policy Optimization for Multi-Turn LLM Agentscited by 0
  22. 2026ABC: Advantage-Based Control Variates for Reinforcement Learning with Verifiable Rewardscited by 0
  23. 2026Where Does Staleness Accumulate? Pool Aware Effective Staleness Control for Asynchronous RL in LLM Post-Trainingcited by 0
  24. 2026Self-Evolving Algorithm-Design Agents: Escaping In-Context Evolutionary Stagnation via Population-Curated Policy Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026ReCo: Reweighting GRPO Against Distributional Concentration
  2. 2025Slow-Fast Policy Optimization: Reposition-Before-Update for LLM Reasoning
  3. 2026RePO: Bridging On-Policy Learning and Off-Policy Knowledge through Rephrasing Policy Optimization
  4. 2025Stabilizing Reinforcement Learning with LLMs: Formulation and Practices
  5. 2025SRPO: A Cross-Domain Implementation of Large-Scale Reinforcement Learning on LLM
  6. 2025TIC-GRPO: Provable and Efficient Optimization for Reinforcement Learning from Human Feedback
  7. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models
  8. 2025Group-Relative REINFORCE Is Secretly an Off-Policy Algorithm: Demystifying Some Myths About GRPO and Its Friends