mlsanityData to 14 Oct 2026

All-Quadrant Bounded Clipping GRPO: Closing the Unbounded Blind Spot for Stable and Generalizable Training

arXiv 2601.03895 ↗First posted 7 Jan 2026cs.LG cs.AI cs.CLcited by 1

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Group Relative Policy Optimization (GRPO) has emerged as a popular algorithm for reinforcement learning with large language models (LLMs). However, GRPO inherits PPO's token-level clipping while replacing token-level advantages with a single sequence-level advantage. Through a four-quadrant analysis of the (likelihood-ratio, advantage) space, we show that this combination leaves one quadrant -- negative advantage combined with an increased likelihood ratio (Q4) -- structurally unbounded, so that a few high-ratio tokens can receive very large suppressive updates that collapse entropy and narrow the reasoning boundary. To address this, we propose All-Quadrant Bounded Clipping GRPO (ABC-GRPO), which applies unconditional clipping in all four quadrants through sign-dependent boundaries. ABC-GRPO clips the likelihood ratio before multiplying by the advantage, adding a trust-region floor in Q2 and a cap in Q4 -- its negative-advantage branch coinciding with dual-clip PPO -- to yield bounded per-step policy displacement in every quadrant. On mathematical reasoning with Qwen3 base models, ABC-GRPO attains the highest Avg@64 and Pass@64: it is statistically superior to GRPO, SAPO, and dual-clip PPO and competitive with the strongest baseline (DAPO), while maintaining substantially higher entropy; the gains transfer to MATH-500 and to out-of-domain code (HumanEval). Ablations isolate Q4 as the dominant blind spot.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 21 of its 27 references that are in the map. First posted on arXiv 7 Jan 2026.

Builds on

21 of 27 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Soft Adaptive Policy Optimization25 Nov
  2. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  3. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  4. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  5. 2023Let's Verify Step by Step31 May
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  7. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 14

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2025Qwen3 Technical Reportcited by 8,820
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  8. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  9. 2017Deep reinforcement learning from human preferencescited by 3,912
  10. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  11. 2015Trust Region Policy Optimizationcited by 2,580
  12. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
2 more
  1. 2025Group Sequence Policy Optimizationcited by 835
  2. 2019Mastering Complex Control in MOBA Games with Deep Reinforcement Learningcited by 161

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Reinforcement Learning for Code Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026How Off-Policy Can GRPO Be? Mu-GRPO for Efficient LLM Reinforcement Learning
  2. 2026Towards Better Training Signal: Advantage Clipped Policy Optimization
  3. 2025SSPO: Subsentence-level Policy Optimization
  4. 2025EDGE-GRPO: Entropy-Driven GRPO with Guided Error Correction for Advantage Diversity
  5. 2025CoRPO: Adding a Correctness Bias to GRPO Improves Generalization
  6. 2026AMIR-GRPO: Inducing Implicit Preference Signals into GRPO
  7. 2026Group Adaptive Clipping Policy Optimization
  8. 2025Sharpness-Guided Group Relative Policy Optimization via Probability Shaping