mlsanityData to 7 Oct 2026

Part I: Tricks or Traps? A Deep Dive into RL for LLM Reasoning

arXiv 2508.08221 ↗First posted 11 Aug 2025cs.LG cs.CLcited by 53

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning for LLM reasoning has rapidly emerged as a prominent research area, marked by a significant surge in related studies on both algorithmic innovations and practical applications. Despite this progress, several critical challenges remain, including the absence of standardized guidelines for employing RL techniques and a fragmented understanding of their underlying mechanisms. Additionally, inconsistent experimental settings, variations in training data, and differences in model initialization have led to conflicting conclusions, obscuring the key characteristics of these techniques and creating confusion among practitioners when selecting appropriate techniques. This paper systematically reviews widely adopted RL techniques through rigorous reproductions and isolated evaluations within a unified open-source framework. We analyze the internal mechanisms, applicable scenarios, and core principles of each technique through fine-grained experiments, including datasets of varying difficulty, model sizes, and architectures. Based on these insights, we present clear guidelines for selecting RL techniques tailored to specific setups, and provide a reliable roadmap for practitioners navigating the RL for the LLM domain. Finally, we reveal that a minimalist combination of two techniques can unlock the learning capability of critic-free policies using vanilla PPO loss. The results demonstrate that our simple combination consistently improves performance, surpassing strategies like GRPO and DAPO.

Key facts. Cited by 53 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 22 of its 33 references that are in the map. First posted on arXiv 11 Aug 2025.

Builds on

22 of 33 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025Reinforcement Learning Optimization for Large-Scale Learning: An Efficient and User-Friendly Scaling Library6 Jun
  2. 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models30 May
  3. 2025SRPO: A Cross-Domain Implementation of Large-Scale Reinforcement Learning on LLM19 Apr
  4. 2025DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning15 Apr
  5. 2025GRPO-LEAD: A Difficulty-Aware Reinforcement Learning Approach for Concise Mathematical Reasoning in Language Models13 Apr
  6. 2025GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning3 Apr
  7. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  8. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  9. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  10. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  11. 2025REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalization4 Jan
  12. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 10

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2024GPT-4o System Cardcited by 4,467
  4. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  5. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 248
  6. 2020Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPOcited by 184
  7. 2025Skywork Open Reasoner 1 Technical Reportcited by 161
  8. 2020What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Studycited by 157
  9. 2024A Comparative Study on Reasoning Patterns of OpenAI's o1 Modelcited by 62
  10. 2024The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarizationcited by 49

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 26; 2026: 27.

53 papers.

Sort:

By area:

  1. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 195
  2. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  3. 2025The Art of Scaling Reinforcement Learning Compute for LLMscited by 91
  4. 2025Jointly Reinforcing Diversity and Quality in Language Model Generationscited by 79
  5. 2025JustRL: Scaling a 1.5B LLM with a Simple RL Recipecited by 57
  6. 2025No Prompt Left Behind: Exploiting Zero-Variance Prompts in LLM Reinforcement Learning via Entropy-Guided Advantage Shapingcited by 48
  7. 2025Demystifying Reinforcement Learning in Agentic Reasoningcited by 32
  8. 2025Single-stream Policy Optimizationcited by 27
  9. 2026KL for a KL: On-Policy Distillation with Control Variate Baselinecited by 24
  10. 2025Attention Illuminates LLM Reasoning: The Preplan-and-Anchor Rhythm Enables Fine-Grained Policy Optimizationcited by 23
  11. 2025SofT-GRPO: Surpassing Discrete-Token LLM Reinforcement Learning via Gumbel-Reparameterized Soft-Thinking Policy Optimizationcited by 18
  12. 2025AgentMath: Empowering Mathematical Reasoning for Large Language Models via Tool-Augmented Agentcited by 17
38 more
  1. 2025Towards High Data Efficiency in Reinforcement Learning with Verifiable Rewardcited by 16
  2. 2025Group-Relative REINFORCE Is Secretly an Off-Policy Algorithm: Demystifying Some Myths About GRPO and Its Friendscited by 13
  3. 2025Native Parallel Reasoner: Reasoning in Parallelism via Self-Distilled Reinforcement Learningcited by 12
  4. 2025DeepSearch: Overcome the Bottleneck of Reinforcement Learning with Verifiable Rewards via Monte Carlo Tree Searchcited by 11
  5. 2025Learn the Ropes, Then Trust the Wins: Self-imitation with Progressive Exploration for Agentic Reinforcement Learningcited by 10
  6. 2025Random Policy Valuation is Enough for LLM Reasoning with Verifiable Rewardscited by 10
  7. 2026F-GRPO: Don't Let Your Policy Learn the Obvious and Forget the Rarecited by 10
  8. 2025Asymmetric Proximal Policy Optimization: mini-critics boost LLM reasoningcited by 9
  9. 2026Group Distributionally Robust Optimization-Driven Reinforcement Learning for LLM Reasoningcited by 9
  10. 2025Towards Understanding Self-play for LLM Reasoningcited by 8
  11. 2026Step Potential Advantage Estimation: Harnessing Intermediate Confidence and Correctness for Efficient Mathematical Reasoningcited by 8
  12. 2026On the Entropy Dynamics in Reinforcement Fine-Tuning of Large Language Modelscited by 8
  13. 2025RAVR: Reference-Answer-guided Variational Reasoning for Large Language Modelscited by 7
  14. 2025TeamPath: Building MultiModal Pathology Experts with Reasoning AI Copilotscited by 5
  15. 2026Segmental Advantage Estimation: Enhancing PPO for Long-Context LLM Trainingcited by 5
  16. 2026Aletheia: What Makes RLVR For Code Verifiers Tick?cited by 4
  17. 2025Retrieval-GRPO: A Multi-Objective Reinforcement Learning Framework for Dense Retrieval in Taobao Searchcited by 3
  18. 2026Probing RLVR training instability through the lens of objective-level hackingcited by 3
  19. 2026DVAO: Dynamic Variance-adaptive Advantage Optimization for Multi-reward Reinforcement Learningcited by 3
  20. 2025T-pro 2.0: An Efficient Russian Hybrid-Reasoning Model and Playgroundcited by 2
  21. 2026Efficient Paths and Dense Rewards: Probabilistic Flow Reasoning for Large Language Modelscited by 2
  22. 2026Balanced Aggregation: Understanding and Fixing Aggregation Bias in GRPOcited by 2
  23. 2026Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learningcited by 2
  24. 2025Future Policy Approximation for Offline Reinforcement Learning in LLM Reasoningcited by 1
  25. 2025Reasoning Palette: Modulating Reasoning via Latent Contextualization for Controllable Exploration for (V)LMscited by 1
  26. 2026Divide-and-Conquer CoT: RL for Reducing Latency via Parallel Reasoningcited by 1
  27. 20263D-RFT: Reinforcement Fine-Tuning for Video-based 3D Scene Understandingcited by 1
  28. 2026Multi-Modal Multi-Agent Reinforcement Learning for Radiology Report Generationcited by 1
  29. 2026Distillation Traps and Guards: A Calibration Knob for LLM Distillabilitycited by 1
  30. 2026Learning Spatiotemporal Sensitivity in Video LLMs via Counterfactual Reinforcement Learningcited by 1
  31. 2026Hyper-ES: Effective Evolution Strategies for LLM Reasoning via Descent Direction Mergingcited by 1
  32. 2026Learning to Solve Hard Problems in RL for LLMs by Never Giving Upcited by 1
  33. 2026Monitoring and Discovering Reward Hacking with Internal Representations during LLM Evaluationscited by 1
  34. 2025GAPO: Robust Advantage Estimation for Real-World Code LLMscited by 0
  35. 2026UCPO: Uncertainty-Aware Policy Optimizationcited by 0
  36. 2026MAPLE: Modality-Aware Post-training and Learning Ecosystemcited by 0
  37. 2026Soft Sequence Policy Optimizationcited by 0
  38. 2026Clipping Bottleneck: Stabilizing RLVR via Stochastic Recovery of Near-Boundary Signalscited by 0

50 most cited of 53 All 53 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspective
  2. 2026Rethinking RL for LLM Reasoning: It's Sparse Policy Selection, Not Capability Learning
  3. 2025Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection
  4. 2025RL of Thoughts: Navigating LLM Reasoning with Inference-time Reinforcement Learning
  5. 2025Accelerating RL for LLM Reasoning with Optimal Advantage Regression
  6. 2025Exploring the Potential of Offline RL for Reasoning in LLMs: A Preliminary Study
  7. 2024Teaching Large Language Models to Reason with Reinforcement Learning
  8. 2025Beyond Accuracy: Dissecting Mathematical Reasoning for LLMs Under Reinforcement Learning