mlsanityData to 7 Oct 2026

Towards Better Training Signal: Advantage Clipped Policy Optimization

arXiv 2609.36816 ↗First posted 29 Sep 2026cs.LGcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning (RL) has become a cornerstone for improving the reasoning capabilities of large language models (LLMs), but the need for on-policy data substantially limits training efficiency. Reusing off-policy data through importance sampling (IS) can improve efficiency but introduce considerable instability. Hence, algorithms such as PPO and GRPO widely adopt IS-ratio clipping to stabilize training. However, training stability and gradient estimate are mainly determined by the product of IS ratio and advantage. To further stabilize training, we propose ACPO, which clips the product of the IS ratio and the advantage, leading to more stable gradient estimates. We also establish a connection between ACPO and gradient clipping in policy mirror descent (PMD), which is a standard technique to stabilize optimization process, and prove the convergence of clipped-PMD under the standard RL setting. Experiments on widely used mathematical reasoning benchmarks show that ACPO consistently outperforms PPO and GRPO in both accuracy and training efficiency, delivering 4-6 percentage points gains on standard math benchmarks, with Qwen3-8B+PPO. Hence, ACPO is a practical and effective alternative to conventional IS-ratio clipping for RL post-training of LLMs.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 14 of its 23 references that are in the map. First posted on arXiv 29 Sep 2026.

Builds on

14 of 23 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Reinforce-Ada: An Adaptive Sampling Framework under Non-linear RL Objectives6 Oct
  2. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
  3. 2025Online Difficulty Filtering for Reasoning Oriented Reinforcement Learning4 Apr
  4. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 10

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  4. 2015Trust Region Policy Optimizationcited by 2,503
  5. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  6. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  7. 2019Mastering Complex Control in MOBA Games with Deep Reinforcement Learningcited by 150
  8. 2021Policy Mirror Descent for Reinforcement Learning: Linear Convergence, New Sampling Complexity, and Generalized Problem Classescited by 149
  9. 2020Mirror Descent Policy Optimizationcited by 102
  10. 2025Prompt Curriculum Learning for Efficient LLM Post-Trainingcited by 40

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025BAPO: Stabilizing Off-Policy Reinforcement Learning for LLMs via Balanced Policy Optimization with Adaptive Clipping
  2. 2025ESPO: Entropy Importance Sampling Policy Optimization
  3. 2026Clipping-Free Policy Optimization for Large Language Models
  4. 2026All-Quadrant Bounded Clipping GRPO: Closing the Unbounded Blind Spot for Stable and Generalizable Training
  5. 2026Ratio-Variance Regularized Policy Optimization for Efficient LLM Fine-tuning
  6. 2025SSPO: Subsentence-level Policy Optimization
  7. 2025DCPO: Dynamic Clipping Policy Optimization
  8. 2026What are Key Factors for Updates in RL for LLM Reasoning?