mlsanityData to 7 Oct 2026

ANO: Robust Policy Optimization via Bounded, Redescending Gain Fields

arXiv 2605.02320 ↗First posted 4 May 2026cs.AI cs.LGcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Proximal Policy Optimization (PPO) dominates reinforcement learning and LLM alignment, yet its hard-clipping mechanism and unconstrained alternatives (e.g., SPO) sit at two extremes of a stability-efficiency dilemma. We argue that this dilemma is best understood dynamically: a surrogate objective is a feedback law on the probability ratio, and its clipping/penalty shape defines a gain field that drives the update dynamics. PPO's clip induces a dead zone (zero feedback outside the trust region), leaving the policy to drift open-loop under momentum; SPO's quadratic penalty induces an unbounded, linearly growing gain that stiffens the dynamics and destabilizes under aggressive step sizes. Guided by this view, we derive Anchored Neighborhood Optimization (ANO), which designs the gain field directly: a C^∞ shaping kernel that anchors the identity map at r=1, peaks exactly at a prescribed trust-region boundary 1+ε, bounds the push on severely off-policy samples by a tunable kappa+, and exerts a bounded, redescending pull of tunable depth kappa- on extreme outliers. The three hyperparameters have decoupled roles, and all internal constants are solved in closed form. Empirically, ANO ranks first on both Atari (40 games) and MuJoCo in IQM and Median of normalized scores. While the runner-up differs across domains (PAPO on Atari, SPO on MuJoCo), ANO is the only method consistently at the top. Under a learning-rate stress test (3×10-4!→!10-3), ANO degrades by only 0.9% whereas PPO collapses by 54.5%, and the stressed ANO still outperforms PPO and PAPO at their best-tuned learning rates.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 22 of its 36 references that are in the map. First posted on arXiv 4 May 2026.

Builds on

22 of 36 references.

References · 22

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2020Learning to summarize from human feedbackcited by 2,562
  5. 2015Trust Region Policy Optimizationcited by 2,503
  6. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  7. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  8. 2023Training Diffusion Models with Reinforcement Learningcited by 939
  9. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 882
  10. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  11. 2021Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learningcited by 395
  12. 2020Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPOcited by 184
10 more
  1. 2017Emergence of Locomotion Behaviours in Rich Environmentscited by 159
  2. 2019Mastering Complex Control in MOBA Games with Deep Reinforcement Learningcited by 150
  3. 2021Tianshou: a Highly Modularized Deep Reinforcement Learning Librarycited by 139
  4. 2019Truly Proximal Policy Optimizationcited by 68
  5. 2020Munchausen Reinforcement Learningcited by 68
  6. 2022EnvPool: A Highly Parallel Reinforcement Learning Environment Execution Enginecited by 58
  7. 2025Hyperspherical Normalization for Scalable Deep Reinforcement Learningcited by 35
  8. 2024Simple Policy Optimizationcited by 31
  9. 2018A Closer Look at Deep Policy Gradientscited by 12
  10. 2023Absolute Policy Optimizationcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Bounded Ratio Reinforcement Learning
  2. 2026Robust Regularized Policy Iteration under Transition Uncertainty
  3. 2025BAPO: Stabilizing Off-Policy Reinforcement Learning for LLMs via Balanced Policy Optimization with Adaptive Clipping
  4. 2026BandPO: Bridging Trust Regions and Ratio Clipping via Probability-Aware Bounds for LLM Reinforcement Learning
  5. 2026Ratio-Variance Regularized Policy Optimization
  6. 2023A dynamical clipping approach with task feedback for Proximal Policy Optimization
  7. 2026PolicyFlow: Policy Optimization with Continuous Normalizing Flow in Reinforcement Learning
  8. 2025Advantage Shaping as Surrogate Reward Maximization: Unifying Pass@K Policy Gradients