mlsanityData to 7 Oct 2026

What's Behind PPO's Collapse in Long-CoT? Value Optimization Holds the Secret

arXiv 2503.01491 ↗First posted 3 Mar 2025cs.LGcited by 95

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning (RL) is pivotal for enabling large language models (LLMs) to generate long chains of thought (CoT) for complex tasks like math and reasoning. However, Proximal Policy Optimization (PPO), effective in many RL scenarios, fails in long CoT tasks. This paper identifies that value initialization bias and reward signal decay are the root causes of PPO's failure. We propose Value-Calibrated PPO (VC-PPO) to address these issues. In VC-PPO, the value model is pretrained to tackle initialization bias, and the Generalized Advantage Estimation (GAE) computation is decoupled between the actor and critic to mitigate reward signal decay. Experiments on the American Invitational Mathematics Examination (AIME) show that VC-PPO significantly boosts PPO performance. Ablation studies show that techniques in VC-PPO are essential in enhancing PPO for long CoT tasks.

Key facts. Cited by 95 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 13 of its 19 references that are in the map. First posted on arXiv 3 Mar 2025.

Builds on

13 of 19 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  2. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 11

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  5. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  6. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  7. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  8. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 956
  9. 2024Step-level Value Preference Optimization for Mathematical Reasoningcited by 71
  10. 2024The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarizationcited by 49
  11. 2024PSYDIAL: Personality-based Synthetic Dialogue Generation using Large Language Modelscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 51; 2026: 44.

95 papers.

Sort:

By area:

  1. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  2. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Examplecited by 288
  3. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 215
  4. 2025UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learningcited by 205
  5. 2025Light-R1: Curriculum SFT, DPO and RL for Long COT from Scratch and Beyondcited by 175
  6. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  7. 2025Seed1.5-Thinking: Advancing Superb Reasoning Models with Reinforcement Learningcited by 159
  8. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoningcited by 158
  9. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 131
  10. 2025Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rolloutscited by 108
  11. 2025The Art of Scaling Reinforcement Learning Compute for LLMscited by 91
  12. 2025S-GRPO: Early Exit via Reinforcement Learning in Reasoning Modelscited by 87
38 more
  1. 2025Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learningcited by 84
  2. 2025Concise Reasoning via Reinforcement Learningcited by 78
  3. 2025Learning When to Think: Shaping Adaptive Reasoning in R1-Style Models via Multi-Stage RLcited by 59
  4. 2025Blending Supervised and Reinforcement Fine-Tuning with Prefix Samplingcited by 57
  5. 2025Reinforcement Learning from Human Feedbackcited by 53
  6. 2025Enigmata: Scaling Logical Reasoning in Large Language Models with Synthetic Verifiable Puzzlescited by 53
  7. 2025Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs?cited by 49
  8. 2025Can Prompt Difficulty be Online Predicted for Accelerating RL Finetuning of Reasoning Models?cited by 48
  9. 2025QwenLong-L1: Towards Long-Context Large Reasoning Models with Reinforcement Learningcited by 46
  10. 2025Reinforced MLLM: A Survey on RL-Based Reasoning in Multimodal Large Language Modelscited by 43
  11. 2025Diversity-Aware Policy Optimization for Large Language Model Reasoningcited by 39
  12. 2025100 Days After DeepSeek-R1: A Survey on Replication Studies and More Directions for Reasoning Language Modelscited by 29
  13. 2025PAG: Multi-Turn Reinforced LLM Self-Correction with Policy as Generative Verifiercited by 28
  14. 2025Turn-PPO: Turn-Level Advantage Estimation with PPO for Improved Multi-Turn RL in Agentic LLMscited by 28
  15. 2026Lightning OPD: Efficient Post-Training for Large Reasoning Models with Offline On-Policy Distillationcited by 28
  16. 2025CAPO: Towards Enhancing LLM Reasoning through Generative Credit Assignmentcited by 25
  17. 2026FIPO: Eliciting Deep Reasoning with Future-KL Influenced Policy Optimizationcited by 23
  18. 2025Generative AI Act II: Test Time Scaling Drives Cognition Engineeringcited by 14
  19. 2025Reinforcement Fine-Tuning Powers Reasoning Capability of Multimodal Large Language Modelscited by 13
  20. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 13
  21. 2025MiroMind-M1: An Open-Source Advancement in Mathematical Reasoning via Context-Aware Multi-Stage Policy Optimizationcited by 12
  22. 2025Trajectory Bellman Residual Minimization: A Simple Value-Based Method for LLM Reasoningcited by 11
  23. 2025R1-RE: Cross-Domain Relation Extraction with RLVRcited by 11
  24. 2025Scheduling Your LLM Reinforcement Learning with Reasoning Treescited by 11
  25. 2025Optimizing Generative Ranking Relevance via Reinforcement Learning in Xiaohongshu Searchcited by 11
  26. 2026Small Generalizable Prompt Predictive Models Can Steer Efficient RL Post-Training of Large Reasoning Modelscited by 11
  27. 2025From Data-Centric to Sample-Centric: Enhancing LLM Reasoning via Progressive Optimizationcited by 10
  28. 2025ReSeek: A Self-Correcting Framework for Search Agents with Instructive Rewardscited by 10
  29. 2025Enhancing Long Chain-of-Thought Reasoning through Multi-Path Plan Aggregationcited by 10
  30. 2026Train at Moving Edge: Online-Verified Prompt Selection for Efficient RL Training of Large Reasoning Modelcited by 10
  31. 2025THOR: Tool-Integrated Hierarchical Optimization via RL for Mathematical Reasoningcited by 9
  32. 2025Asymmetric Proximal Policy Optimization: mini-critics boost LLM reasoningcited by 9
  33. 2025VAR-MATH: Probing True Mathematical Reasoning in LLMS via Symbolic Multi-Instance Benchmarkscited by 8
  34. 2025Stabilizing Off-Policy Training for Long-Horizon LLM Agent via Turn-Level Importance Sampling and Clipping-Triggered Normalizationcited by 8
  35. 2026Dynamics-Predictive Sampling for Active RL Finetuning of Large Reasoning Modelscited by 8
  36. 2026CompactionRL: Reinforcement Learning with Context Compaction for Long-Horizon Agentscited by 8
  37. 2026V0: A Generalist Value Model for Any Policy at State Zerocited by 7
  38. 2026SPPO: Sequence-Level PPO for Long-Horizon Reasoning Taskscited by 7

50 most cited of 95 All 95 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Truncated Proximal Policy Optimization
  2. 2026Start Classifying: Categorical Critics for LLM Reinforcement Learning
  3. 2024VinePPO: Refining Credit Assignment in RL Training of LLMs
  4. 2026DISPO: Enhancing Training Efficiency and Stability in Reinforcement Learning for Large Language Model Mathematical Reasoning
  5. 2025VRPO: Rethinking Value Modeling for Robust RL under Noisy Supervision in LLM Post-Training
  6. 2023Proximal Policy Optimization Actual Combat: Manipulating Output Tokenizer Length
  7. 2025On-Policy RL with Optimal Reward Baseline
  8. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study