mlsanityData to 14 Oct 2026

OR Else: A Differentiable Trust Region for Policy Optimization

arXiv 2607.18163 ↗First posted 20 Jul 2026cs.LG cs.AIcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

PPO and the GRPO baseline studied here use clipped surrogate objectives whose favorable-direction saturation introduces an abrupt change in the scalar objective's derivative. We ask whether Output Reset (OR), a smooth one-sided saturation rule, offers a useful alternative for large language model post-training. PPO-OR and GRPO-OR replace the clipped policy term with an OR squared-margin loss in rollout-relative token log-ratio space; the advantage sign determines the update direction, and a token contributes zero direct OR residual after crossing the favorable margin. We compare PPO-clip with PPO-OR under generalized advantage estimation (GAE), and GRPO with GRPO-OR under group-relative advantages, using textttLlama-3.2-1B-Instruct on Anthropic texttthh-rlhf with one shared reward model and three seeds per method. Under GAE, PPO-OR has a mean final training-time reward-model score 0.305 higher than PPO-clip, with a larger observed across-seed spread. Under group-relative advantages, GRPO-OR does not have a higher mean score, but shows a smaller observed spread, a near-zero terminal OR residual, and a declining overshoot fraction, while the matched GRPO clipped-objective trace remains variable. Both group-relative methods exhibit substantially larger rollout-to-current log-ratio displacement than the GAE methods, and OR does not consistently reduce it. Thus, OR changes optimization behavior in both matched comparisons, but the observed reward effect differs between them. At G=2, the GRPO-OR diagnostics do not translate into a reward-score gain. Whether larger groups change this outcome remains open. The reported scores are training-time reward-model measurements, not held-out human-preference performance.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 19 of its 29 references that are in the map. First posted on arXiv 20 Jul 2026.

Builds on

19 of 29 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  2. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  4. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 15

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2017Deep reinforcement learning from human preferencescited by 3,912
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  5. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  6. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  7. 2020Learning to summarize from human feedbackcited by 2,588
  8. 2015Trust Region Policy Optimizationcited by 2,580
  9. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  10. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  11. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 187
  12. 2019Truly Proximal Policy Optimizationcited by 69
3 more
  1. 2024Simple Policy Optimizationcited by 32
  2. 2021On Proximal Policy Optimization's Heavy-tailed Gradientscited by 31
  3. 2024Making Sigmoid-MSE Great Again: Output Reset Challenges Softmax Cross-Entropy in Neural Network Classificationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025BAPO: Stabilizing Off-Policy Reinforcement Learning for LLMs via Balanced Policy Optimization with Adaptive Clipping
  2. 2026A First-Principles Derivation of LLM Policy Optimization: From Expected Reward to GRPO and Its Structural Extensions
  3. 2025CoRPO: Adding a Correctness Bias to GRPO Improves Generalization
  4. 2025Group-Relative REINFORCE Is Secretly an Off-Policy Algorithm: Demystifying Some Myths About GRPO and Its Friends
  5. 2026EasyPPO: Stabilizing the Critic Is Key
  6. 2026Range-GRPO: Policy Optimization via Pairwise Relations among Reward Intervals
  7. 2026Stabilizing Reinforcement Learning for Diffusion Language Models
  8. 2026BV-Blend: Uncertainty-Weighted Historical Baselines for Stable Critic-Free RL with Verifiable Rewards