mlsanityData to 14 Oct 2026

SRPO: A Cross-Domain Implementation of Large-Scale Reinforcement Learning on LLM

arXiv 2504.14286 ↗First posted 19 Apr 2025cs.LGcited by 68

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Recent advances of reasoning models, exemplified by OpenAI's o1 and DeepSeek's R1, highlight the significant potential of Reinforcement Learning (RL) to enhance the reasoning capabilities of Large Language Models (LLMs). However, replicating these advancements across diverse domains remains challenging due to limited methodological transparency. In this work, we present two-Staged history-Resampling Policy Optimization (SRPO), which surpasses the performance of DeepSeek-R1-Zero-32B on the AIME24 and LiveCodeBench benchmarks. SRPO achieves this using the same base model as DeepSeek (i.e. Qwen2.5-32B), using only about 1/10 of the training steps required by DeepSeek-R1-Zero-32B, demonstrating superior efficiency. Building upon Group Relative Policy Optimization (GRPO), we introduce two key methodological innovations: (1) a two-stage cross-domain training paradigm designed to balance the development of mathematical reasoning and coding proficiency, and (2) History Resampling (HR), a technique to address ineffective samples. Our comprehensive experiments validate the effectiveness of our approach, offering valuable insights into scaling LLM reasoning capabilities across diverse tasks.

Key facts. Cited by 68 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 14 of its 20 references that are in the map. First posted on arXiv 19 Apr 2025.

Builds on

14 of 20 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  2. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  3. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan

Other references · 11

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2024Qwen2.5 Technical Reportcited by 4,942
  4. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  5. 2016Concrete Problems in AI Safetycited by 1,642
  6. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  7. 2020Curriculum Learning for Reinforcement Learning Domains: A Framework and Surveycited by 373
  8. 2025Demystifying Long Chain-of-Thought Reasoning in LLMscited by 326
  9. 2019Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspectivecited by 108
  10. 2025An Empirical Study on Eliciting and Improving R1-like Reasoning Modelscited by 78
  11. 2017Reinforcement Learning with a Corrupted Reward Channelcited by 70

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 38; 2026: 30.

68 papers.

Sort:

By area:

  1. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Examplecited by 295
  2. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
  3. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoningcited by 162
  4. 2025Geometric-Mean Policy Optimizationcited by 111
  5. 2025SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy Optimizationcited by 89
  6. 2025Part I: Tricks or Traps? A Deep Dive into RL for LLM Reasoningcited by 56
  7. 2025The Path Not Taken: RLVR Provably Learns Off the Principalscited by 56
  8. 2025Can Prompt Difficulty be Online Predicted for Accelerating RL Finetuning of Reasoning Models?cited by 52
  9. 2025rStar-Coder: Scaling Competitive Code Reasoning with a Large-Scale Verified Datasetcited by 51
  10. 2025SynLogic: Synthesizing Verifiable Reasoning Data at Scale for Learning Logical Reasoning and Beyondcited by 50
  11. 2025First Return, Entropy-Eliciting Explorecited by 45
  12. 2025Phi-4-Mini-Reasoning: Exploring the Limits of Small Reasoning Language Models in Mathcited by 44
38 more
  1. 2025Diversity-Aware Policy Optimization for Large Language Model Reasoningcited by 41
  2. 2025FlowRL: Matching Reward Distributions for LLM Reasoningcited by 40
  3. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecyclecited by 35
  4. 2025100 Days After DeepSeek-R1: A Survey on Replication Studies and More Directions for Reasoning Language Modelscited by 30
  5. 2025TimeOmni-1: Incentivizing Complex Reasoning with Time Series in Large Language Modelscited by 27
  6. 2026Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statisticcited by 21
  7. 2025Plan Then Action:High-Level Planning Guidance Reinforcement Learning for LLM Reasoningcited by 19
  8. 2025CURE: Critical-Token-Guided Re-Concatenation for Entropy-Collapse Preventioncited by 18
  9. 2025DisCO: Reinforcing Large Reasoning Models with Discriminative Constrained Optimizationcited by 16
  10. 2026Small Generalizable Prompt Predictive Models Can Steer Efficient RL Post-Training of Large Reasoning Modelscited by 14
  11. 2025KAT-V1: Kwai-AutoThink Technical Reportcited by 12
  12. 2025Scheduling Your LLM Reinforcement Learning with Reasoning Treescited by 12
  13. 2025AdapThink: Adaptive Thinking Preferences for Reasoning Language Modelcited by 11
  14. 2025Enhancing Long Chain-of-Thought Reasoning through Multi-Path Plan Aggregationcited by 11
  15. 2025Supervised Reinforcement Learning: From Expert Trajectories to Step-wise Reasoningcited by 11
  16. 2026Dynamics-Predictive Sampling for Active RL Finetuning of Large Reasoning Modelscited by 11
  17. 2025Ring-lite: Scalable Reasoning via C3PO-Stabilized Reinforcement Learning for LLMscited by 10
  18. 2025From Data-Centric to Sample-Centric: Enhancing LLM Reasoning via Progressive Optimizationcited by 10
  19. 2026Train at Moving Edge: Online-Verified Prompt Selection for Efficient RL Training of Large Reasoning Modelcited by 10
  20. 2026TRACE: A Unified Rollout Budget Allocation Framework for Efficient Agentic Reinforcement Learningcited by 10
  21. 2026RLVR without Ineffective Samples: Group Prioritized Off-Policy Optimization for LLM Reasoningcited by 9
  22. 2025VAR-MATH: Probing True Mathematical Reasoning in LLMS via Symbolic Multi-Instance Benchmarkscited by 8
  23. 2026Buffer Matters: Unleashing the Power of Off-Policy Reinforcement Learning in Large Language Model Reasoningcited by 8
  24. 2025DISCO Balances the Scales: Adaptive Domain- and Difficulty-Aware Reinforcement Learning on Imbalanced Datacited by 7
  25. 2025Reinforcement Learning for Out-of-Distribution Reasoning in LLMs: An Empirical Study on Diagnosis-Related Group Codingcited by 7
  26. 2025Explore Data Left Behind in Reinforcement Learning for Reasoning Language Modelscited by 7
  27. 2025Unearthing Gems from Stones: Policy Optimization with Negative Sample Augmentation for LLM Reasoningcited by 6
  28. 2025RuleReasoner: Reinforced Rule-based Reasoning via Domain-aware Dynamic Samplingcited by 6
  29. 2026Collaborative Multi-Agent Scripts Generation for Enhancing Imperfect-Information Reasoning in Murder Mystery Gamescited by 6
  30. 2025CodeReasoner: Enhancing the Code Reasoning Ability with Reinforcement Learningcited by 4
  31. 2026Rewards as Labels: Revisiting RLVR from a Classification Perspectivecited by 4
  32. 2026LACONIC: Length-Aware Constrained Reinforcement Learning for LLMcited by 4
  33. 2025The Challenge of Teaching Reasoning to LLMs Without RL or Distillationcited by 3
  34. 2026Resource-Efficient Reinforcement for Reasoning Large Language Models via Dynamic One-Shot Policy Refinementcited by 3
  35. 2025FinLMM-R1: Enhancing Financial Reasoning in LMM through Scalable Data and Reward Designcited by 2
  36. 2025GCPO: When Contrast Fails, Go Goldcited by 2
  37. 2025What Makes Reasoning Invalid: Echo Reflection Mitigation for Large Language Modelscited by 2
  38. 2026One Sample to Rule Them All: Extreme Data Efficiency in Multidiscipline Reasoning with Reinforcement Learningcited by 2

50 most cited of 68 All 68 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Slow-Fast Policy Optimization: Reposition-Before-Update for LLM Reasoning
  2. 2025A Survey of Reinforcement Learning for Large Reasoning Models
  3. 2025SRPO: Enhancing Multimodal LLM Reasoning via Reflection-Aware Reinforcement Learning
  4. 2025SSPO: Subsentence-level Policy Optimization
  5. 2025Scaf-GRPO: Scaffolded Group Relative Policy Optimization for Enhancing LLM Reasoning
  6. 2025Squeeze the Soaked Sponge: Efficient Off-policy Reinforcement Finetuning for Large Language Model
  7. 2025ScRPO: From Errors to Insights
  8. 2026Exploration-Driven Optimization for Test-Time Large Language Model Reasoning