mlsanityData to 7 Oct 2026

Clipping-Free Policy Optimization for Large Language Models

arXiv 2601.22801 ↗First posted 30 Jan 2026cs.LGcited by 3

On the map: LLM alignment / RLHF / preference optimisation → Process reward models for reasoning

Abstract (arXiv)

Reinforcement learning has become central to post-training large language models, yet dominant algorithms rely on clipping mechanisms that introduce optimization issues at scale, including zero-gradient regions, reward hacking, and training instability. We propose Clipping-Free Policy Optimization (CFPO), which replaces heuristic clipping with a convex quadratic penalty derived from Total Variation divergence constraints, yielding an everywhere-differentiable objective that enforces stable policy updates without hard boundaries. We evaluate CFPO across both reasoning and alignment settings. In reasoning, CFPO matches clipping-based methods on downstream benchmarks while extending the stable training regime. In alignment, CFPO mitigates verbosity exploitation and reduces capability degradation, while achieving competitive instruction-following performance. CFPO requires only a one-line code change and no additional hyperparameters. Our results suggest that CFPO is a promising drop-in alternative to clipping-based methods for LLM post-training.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Process reward models for reasoning, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 36 of its 52 references that are in the map. First posted on arXiv 30 Jan 2026.

Builds on

36 of 52 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025λ-GRPO: Unifying the GRPO Frameworks with Learnable Token Preferences8 Oct

Other references · 35

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  7. 2024Qwen2.5 Technical Reportcited by 4,713
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  10. 2017Deep reinforcement learning from human preferencescited by 3,846
  11. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  12. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
23 more
  1. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  2. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  3. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  4. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  5. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
  6. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  7. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  8. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  9. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  10. 2017Constrained Policy Optimizationcited by 928
  11. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  12. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  13. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  14. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  15. 2018Maximum a Posteriori Policy Optimisationcited by 318
  16. 2020Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPOcited by 184
  17. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 184
  18. 2021Policy Mirror Descent for Reinforcement Learning: Linear Convergence, New Sampling Complexity, and Generalized Problem Classescited by 149
  19. 2020Mirror Descent Policy Optimizationcited by 102
  20. 2019Truly Proximal Policy Optimizationcited by 68
  21. 2021Generalized Proximal Policy Optimization with Sample Reusecited by 39
  22. 2025Low-probability Tokens Sustain Exploration in Reinforcement Learning with Verifiable Rewardcited by 25
  23. 2017f-Divergence constrained policy improvementcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Turning Off-Policy Tokens On-Policy: A Plug-in Approach for Improving LLM Alignmentcited by 0
  2. 2026Deconstructing Off-Policy Ratios: Entropy-Normalized Trust Regions for Asynchronous Reinforcement Learningcited by 0
  3. 2026BRACE: Anchored Bellman-Residual Correction for Stale Critics in Asynchronous RLcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025BAPO: Stabilizing Off-Policy Reinforcement Learning for LLMs via Balanced Policy Optimization with Adaptive Clipping
  2. 2026Ratio-Variance Regularized Policy Optimization for Efficient LLM Fine-tuning
  3. 2026Rethinking the Trust Region in LLM Reinforcement Learning
  4. 2026Towards Better Training Signal: Advantage Clipped Policy Optimization
  5. 2026From log π to π: Taming Divergence in Soft Clipping via Bilateral Decoupled Decay of Probability Gradient Weight
  6. 2025M-GRPO: Stabilizing Self-Supervised Reinforcement Learning for Large Language Models with Momentum-Anchored Policy Optimization
  7. 2025CPGD: Toward Stable Rule-based Reinforcement Learning for Language Models
  8. 2026Buffer Matters: Unleashing the Power of Off-Policy Reinforcement Learning in Large Language Model Reasoning