mlsanityData to 7 Oct 2026

CPPO: Accelerating the Training of Group Relative Policy Optimization-Based Reasoning Models

arXiv 2503.22342 ↗First posted 28 Mar 2025cs.AIcited by 93

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

This paper introduces Completion Pruning Policy Optimization (CPPO) to accelerate the training of reasoning models based on Group Relative Policy Optimization (GRPO). GRPO, while effective, incurs high training costs due to the need to sample multiple completions for each question. Our experiment and theoretical analysis reveal that the number of completions impacts model accuracy yet increases training time multiplicatively, and not all completions contribute equally to policy training -- their contribution depends on their relative advantage. To address these issues, we propose CPPO, which prunes completions with low absolute advantages, significantly reducing the number needed for gradient calculation and updates. Additionally, we introduce a dynamic completion allocation strategy to maximize GPU utilization by incorporating additional questions, further enhancing training efficiency. Experiments show that CPPO achieves up to 7.98× speedup on GSM8K and 3.48× on Math while preserving or even enhancing the accuracy compared to the original GRPO. We release our code at hrefhttps://github.com/lzhxmu/CPPOhttps://github.com/lzhxmu/CPPO.

Key facts. Cited by 93 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 20 of its 26 references that are in the map. First posted on arXiv 28 Mar 2025.

Builds on

20 of 26 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  2. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  3. 2025Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learning20 Feb
  4. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  5. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 15

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  5. 2024Qwen2.5 Technical Reportcited by 4,713
  6. 2023Qwen Technical Reportcited by 3,273
  7. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  8. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  9. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  10. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  11. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  12. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 564
3 more
  1. 2025rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinkingcited by 308
  2. 2025TokenSkip: Controllable Chain-of-Thought Compression in LLMscited by 246
  3. 2024C3oT: Generating Shorter Chain-of-Thought without Compromising Effectivenesscited by 117

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 44; 2026: 49.

93 papers.

Sort:

By area:

  1. 2025Group-in-Group Policy Optimization for LLM Agent Trainingcited by 442
  2. 2025Geometric-Mean Policy Optimizationcited by 107
  3. 2025A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibilitycited by 89
  4. 2025SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy Optimizationcited by 87
  5. 2025Segment Policy Optimization: Effective Segment-Level Credit Assignment in RL for Large Language Modelscited by 75
  6. 2025BREAD: Branched Rollouts from Expert Anchors Bridge SFT & RL for Reasoningcited by 57
  7. 2026Hindsight Credit Assignment for Long-Horizon LLM Agentscited by 51
  8. 2025Can Prompt Difficulty be Online Predicted for Accelerating RL Finetuning of Reasoning Models?cited by 48
  9. 2025Reinforced MLLM: A Survey on RL-Based Reasoning in Multimodal Large Language Modelscited by 43
  10. 2025RePO: Replay-Enhanced Policy Optimizationcited by 36
  11. 2025OctoNav: Towards Generalist Embodied Navigationcited by 33
  12. 2025Delving into RL for Image Generation with CoT: A Study on DPO vs. GRPOcited by 31
38 more
  1. 2025SPEED-RL: Faster Training of Reasoning Models via Online Curriculum Learningcited by 27
  2. 2026Beyond Trajectory-Level Attribution: Graph-Based Credit Assignment for Agentic Reinforcement Learningcited by 23
  3. 2025MO-GRPO: Mitigating Reward Hacking of Group Relative Policy Optimization on Multi-Objective Problemscited by 21
  4. 2025Co-rewarding: Stable Self-supervised RL for Eliciting Reasoning in Large Language Modelscited by 20
  5. 2026Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statisticcited by 19
  6. 2025DisCO: Reinforcing Large Reasoning Models with Discriminative Constrained Optimizationcited by 16
  7. 2025CurES: From Gradient Analysis to Efficient Curriculum Learning for Reasoning LLMscited by 16
  8. 2026Adaptive Rollout Allocation for Online Reinforcement Learning with Verifiable Rewardscited by 15
  9. 2025cadrille: Multi-modal CAD Reconstruction with Reinforcement Learningcited by 14
  10. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 14
  11. 2025EFRame: Deeper Reasoning via Exploration-Filter-Replay Reinforcement Learning Frameworkcited by 13
  12. 2025Don't Tell the Answer, Truly Guide the Reasoning During RL Rolloutscited by 13
  13. 2026Train Less, Learn More: Adaptive Efficient Rollout Optimization for Group-Based Reinforcement Learningcited by 13
  14. 2025Trajectory Bellman Residual Minimization: A Simple Value-Based Method for LLM Reasoningcited by 11
  15. 2025Done Is Better than Perfect: Unlocking Efficient Reasoning by Structured Multi-Turn Decompositioncited by 11
  16. 2025SPEC-RL: Accelerating On-Policy Reinforcement Learning with Speculative Rolloutscited by 11
  17. 2026Small Generalizable Prompt Predictive Models Can Steer Efficient RL Post-Training of Large Reasoning Modelscited by 11
  18. 2025GeometryZero: Advancing Geometry Solving via Group Contrastive Policy Optimizationcited by 9
  19. 2026TRACE: A Unified Rollout Budget Allocation Framework for Efficient Agentic Reinforcement Learningcited by 9
  20. 2025LearnAlign: Data Selection for LLM Reinforcement Learning with Improved Gradient Alignmentcited by 8
  21. 2026Prune as You Generate: Online Rollout Pruning for Faster and Better RLVRcited by 8
  22. 2026CAPO: Critic-Guided Action-Aligned Policy Optimization for Advancing LLM Agent Capabilitiescited by 8
  23. 2025Reinforcement Learning for Out-of-Distribution Reasoning in LLMs: An Empirical Study on Diagnosis-Related Group Codingcited by 7
  24. 2025FastGRPO: Accelerating Policy Optimization via Concurrency-aware Speculative Decoding and Online Draft Learningcited by 7
  25. 2025Language Model Based Text-to-Audio Generation: Anti-Causally Aligned Collaborative Residual Transformerscited by 7
  26. 2026LaDi-RL: Latent Diffusion Reasoning Prevents Entropy Collapse in Reinforcement Learningcited by 7
  27. 2026SPPO: Sequence-Level PPO for Long-Horizon Reasoning Taskscited by 7
  28. 2025AutoChemSchematic AI: Agentic Physics-Aware Automation for Chemical Manufacturing Scale-Upcited by 6
  29. 2026MC-GRPO: Median-Centered Group Relative Policy Optimization for Small-Rollout Reinforcement Learningcited by 6
  30. 2025Dissecting Long-Chain-of-Thought Reasoning Models: An Empirical Studycited by 5
  31. 2025MOA: Multi-Objective Alignment for Role-Playing Agentscited by 5
  32. 2026StaRPO: Stability-Augmented Reinforcement Policy Optimizationcited by 5
  33. 2026Spend Your Rollouts Where It Counts: Rollout Allocation for Group-Based RL Post-Trainingcited by 5
  34. 2025Enhancing Large Language Models through Structured Reasoningcited by 4
  35. 2025Bridging On-Device and Cloud LLMs for Collaborative Reasoning: A Unified Methodology for Local Routing and Post-Trainingcited by 4
  36. 2025Interactive Learning for LLM Reasoningcited by 4
  37. 2025ToolExpander: Extending the Frontiers of Tool-Using Reinforcement Learning to Weak LLMscited by 4
  38. 2025PuzzleCraft: Exploration-Aware Curriculum Learning for Puzzle-Based RLVR in VLMscited by 4

50 most cited of 93 All 93 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026BPPO: Binary Prefix Policy Optimization for Efficient GRPO-Style Reasoning RL with Concise Responses
  2. 2026Unbiased Dynamic Pruning for Efficient Group-Based Policy Optimization
  3. 2025Scaf-GRPO: Scaffolded Group Relative Policy Optimization for Enhancing LLM Reasoning
  4. 2026CoDistill-GRPO: A Co-Distillation Recipe for Efficient Group Relative Policy Optimization
  5. 2026GRPO Training Dynamics for Small Language Models
  6. 2026GRPO-VPS: Enhancing Group Relative Policy Optimization with Verifiable Process Supervision for Effective Reasoning
  7. 2026PrAg-PO: Prompt Augmented Policy Optimization for Robust and Diverse Mathematical Reasoning
  8. 2026MMR-GRPO: Accelerating GRPO-Style Training through Diversity-Aware Reward Reweighting