mlsanityData to 14 Oct 2026

ExGRPO: Learning to Reason from Experience

arXiv 2510.02245 ↗First posted 2 Oct 2025cs.LG cs.AI cs.CLcited by 53

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning from verifiable rewards (RLVR) is an emerging paradigm for improving the reasoning ability of large language models. However, standard on-policy training discards rollout experiences after a single update, leading to computational inefficiency and instability. While prior work on RL has highlighted the benefits of reusing past experience, the role of experience characteristics in shaping learning dynamics of large reasoning models remains underexplored. In this paper, we are the first to investigate what makes a reasoning experience valuable and identify rollout correctness and entropy as effective indicators of experience value. Based on these insights, we propose ExGRPO (Experiential Group Relative Policy Optimization), a framework that organizes and prioritizes valuable experiences, and employs a mixed-policy objective to balance exploration with experience exploitation. Experiments on five backbone models (1.5B-8B parameters) show that ExGRPO consistently improves reasoning performance on mathematical/general benchmarks, with an average gain of +3.5/7.6 points over on-policy RLVR. Moreover, ExGRPO stabilizes training on both stronger and weaker models where on-policy methods fail. These results highlight principled experience management as a key ingredient for efficient and scalable RLVR.

Key facts. Cited by 53 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 53 of its 62 references that are in the map. First posted on arXiv 2 Oct 2025.

Builds on

53 of 62 references.

Closest ancestors · 26

Same direction, newest first

  1. 2025Language Models that Think, Chat Better24 Sep
  2. 2025A Survey of Reinforcement Learning for Large Reasoning Models10 Sep
  3. 2025RLEP: Reinforcement Learning with Experience Replay for LLM Reasoning10 Jul
  4. 2025Squeeze the Soaked Sponge: Efficient Off-policy Reinforcement Finetuning for Large Language Model9 Jul
  5. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs17 Jun
  6. 2025RePO: Replay-Enhanced Policy Optimization11 Jun
  7. 2025Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questions9 Jun
  8. 2025KDRL: Post-Training Reasoning LLMs via Unified Knowledge Distillation and Reinforcement Learning2 Jun
  9. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning2 Jun
  10. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
  11. 2025General-Reasoner: Advancing LLM Reasoning Across All Domains20 May
  12. 2025SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy Optimization18 May
14 more
  1. 2025Learning to Reason under Off-Policy Guidance21 Apr
  2. 2025Improving RL Exploration for LLM Reasoning through Retrospective Replay19 Apr
  3. 2025A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforce15 Apr
  4. 2025GRPO-LEAD: A Difficulty-Aware Reinforcement Learning Approach for Concise Mathematical Reasoning in Language Models13 Apr
  5. 2025GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning3 Apr
  6. 2025Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across Diverse Domains31 Mar
  7. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  8. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  9. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  10. 2025Process Reinforcement through Implicit Rewards3 Feb
  11. 2024Free Process Rewards without Process Labels2 Dec
  12. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Training22 Nov
  13. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs24 Oct
  14. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 27

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  6. 2015Continuous control with deep reinforcement learningcited by 3,407
  7. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  8. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  9. 2015Trust Region Policy Optimizationcited by 2,580
  10. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  11. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  12. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
15 more
  1. 2015Prioritized Experience Replaycited by 1,099
  2. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  3. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  4. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 629
  5. 2016Sample Efficient Actor-Critic with Experience Replaycited by 188
  6. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 132
  7. 2025On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weightingcited by 91
  8. 2025Agentic AI for Scientific Discovery: A Survey of Progress, Challenges, and Future Directionscited by 77
  9. 2025ARPO:End-to-End Policy Optimization for GUI Agents with Experience Replaycited by 77
  10. 2025Tapered Off-Policy REINFORCE: Stable and efficient reinforcement learning for LLMscited by 61
  11. 2025xVerify: Efficient Answer Verifier for Reasoning Model Evaluationscited by 55
  12. 2025Asymmetric REINFORCE for off-Policy Reinforcement Learning: Balancing positive and negative rewardscited by 29
  13. 2020Statistically Efficient Off-Policy Policy Gradientscited by 25
  14. 2022Prioritizing Samples in Reinforcement Learning with Reducible Losscited by 22
  15. 2025Soft Policy Optimization: Online Off-Policy RL for Sequence Modelscited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 50.

53 papers.

Sort:

By area:

  1. 2026Skill1: Unified Evolution of Skill-Augmented Agents via Reinforcement Learningcited by 35
  2. 2026Contextual Rollout Bandits for Reinforcement Learning with Verifiable Rewardscited by 11
  3. 2026Understanding Diversity Collapse in RLVR via the Lens of Overtrainingcited by 10
  4. 2026RLVR without Ineffective Samples: Group Prioritized Off-Policy Optimization for LLM Reasoningcited by 9
  5. 2026Code-A1: Adversarial Evolving of Code LLM and Test LLM via Reinforcement Learningcited by 8
  6. 2026Prompt replay: speeding up grpo with on-policy reuse of high-signal promptscited by 8
  7. 2025AWPO: Enhancing Tool-Use of Large Language Models through Adaptive Integration of Reasoning Rewardscited by 7
  8. 2026Test-time Recursive Thinking: Self-Improvement without External Feedbackcited by 7
  9. 2026From Conflict to Consensus: Boosting Medical Reasoning via Multi-Round Agentic RAGcited by 7
  10. 2026Why Multi-Step Tool-Use Reinforcement Learning Collapses and How Supervisory Signals Fix Itcited by 7
  11. 2026Off-Context GRPO: Learning to Reason on Hard Problems using Privileged Informationcited by 7
  12. 2026Gradient Starvation in Binary-Reward GRPO: Why Group-Mean Centering Fails and Why the Simplest Fix Workscited by 6
38 more
  1. 2026Characterizing, Evaluating, and Optimizing Complex Reasoningcited by 5
  2. 2026Complementary RL: Towards Efficient Experience-Driven Agent Learningcited by 5
  3. 2026Beyond Uniform Credit Assignment: Selective Eligibility Traces for RLVRcited by 5
  4. 2025Putting on the Thinking Hats: A Survey on Chain of Thought Fine-tuning from the Perspective of Human Reasoning Mechanismcited by 4
  5. 2026DyJR: Preserving Diversity in Reinforcement Learning with Verifiable Rewards via Dynamic Jensen-Shannon Replaycited by 4
  6. 2026Training Reasoning Models on Saturated Problems via Failure-Prefix Conditioningcited by 3
  7. 2026P^2O: Joint Policy and Prompt Optimizationcited by 3
  8. 2026Joint Optimization of Reasoning and Dual-Memory for Self-Learning Diagnostic Agentcited by 3
  9. 2026Find, Fix, Reason: Context Repair for Video Reasoningcited by 3
  10. 2026Mem-π: Adaptive Memory through Learning When and What to Generatecited by 3
  11. 2026Zone of Proximal Policy Optimization: Teacher in Prompts, Not Gradientscited by 3
  12. 2026TRACE: A Self-Evolving Skill Bank for Consistent, Limit-Aware LLM Agentscited by 3
  13. 2026New Skills or Sharper Primitives? A Probabilistic Perspective on the Emergence of Reasoning in RLVRcited by 2
  14. 2026Towards Effective Experiential Learning: Dual Guidance for Utilization and Internalizationcited by 2
  15. 2026Can LLMs Learn to Reason Robustly under Noisy Supervision?cited by 2
  16. 2026Rollout-Level Advantage-Prioritized Experience Replay for GRPOcited by 2
  17. 2026Learning at the Right Pace: Adaptive Data Scheduling Improves LLM Reinforcement Learningcited by 2
  18. 2026DRIFT: Difficulty Routing Self-DIstillation with Rhythm-Gated Exploration and Success BuFfer Trainingcited by 2
  19. 2026Experience Augmented Policy Optimization for LLM Reasoningcited by 2
  20. 2026EDGE: Experience-Distillation for Guided Exploration in Agentic Reinforcement Learningcited by 2
  21. 2026Transformation-Augmented GRPO for Enhancing Exploration in Reasoning of Large Language Modelscited by 1
  22. 2026Seirenes: Adversarial Self-Play with Evolving Distractions for LLM Reasoningcited by 1
  23. 2026BASIS: Batchwise Advantage Estimation from Single-Rollout Information Sharing for LLM Reasoningcited by 1
  24. 2026Mechanistically Interpreting the Role of Sample Difficulty in RLVR for LLMscited by 1
  25. 2026Learning to Solve, Forgetting to Retain: Correct-Set Turnover in RLVRcited by 1
  26. 2026A First-Principles Derivation of LLM Policy Optimization: From Expected Reward to GRPO and Its Structural Extensionscited by 1
  27. 2026The Mirage of Optimizing Training Policies: Monotonic Inference Policies as the Real Objective for LLM Reinforcement Learningcited by 1
  28. 2026Headroom-Drift Replay: A Primitive for Principled Replay Control in GRPOcited by 1
  29. 2025Rewarding Structural Conformance of Reasoning using Process Miningcited by 0
  30. 2026Less Noise, More Voice: Reinforcement Learning for Reasoning via Instruction Purificationcited by 0
  31. 2026OGER: A Robust Offline-Guided Exploration Reward for Hybrid Reinforcement Learningcited by 0
  32. 2026When to Stop Reusing: Dynamic Gradient Gating for Sample-Efficient RLVRcited by 0
  33. 2026Does Seeing More Mean Knowing More? Mono-Anchored Advantage Normalization for Multi-Source Visual Reasoningcited by 0
  34. 2026Tandem Reinforcement Learning with Verifiable Rewardscited by 0
  35. 2026ARMOR: Stabilizing On-Policy LLM RL with Off-Policy Anchor Samplescited by 0
  36. 2026ARC: Fair Relative Advantage Comparison in Open-Ended Real-World Interactioncited by 0
  37. 2026CARE: Causally-Aligned Reasoning Exploration for Medical Large Language Modelscited by 0
  38. 2026Reinforcement Learning with Verifiable Rewards for Small Search Agentscited by 0

50 most cited of 53 All 53 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Explore Data Left Behind in Reinforcement Learning for Reasoning Language Models
  2. 2025EEPO: Exploration-Enhanced Policy Optimization via Sample-Then-Forget
  3. 2025Towards High Data Efficiency in Reinforcement Learning with Verifiable Reward
  4. 2025FAPO: Flawed-Aware Policy Optimization for Efficient and Reliable Reasoning
  5. 2026Revisiting Reinforcement Learning with Verifiable Rewards from a Contrastive Perspective
  6. 2026Learning from Failures: Correction-Oriented Policy Optimization with Verifiable Rewards
  7. 2025Reasoning with Exploration: An Entropy Perspective
  8. 2026Decoupling Exploration from Optimization in RLVR