mlsanityData to 7 Oct 2026

HPO: Hysteretic Policy Optimization for Stable and Efficient Training under Sparse-Reward Regime

arXiv 2605.30201 ↗First posted 28 May 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

We investigate a narrow but common failure mode of GRPO-style reinforcement learning in the context of sparse verifiable rewards: early updates contain more responses with negative advantages than those with positive advantages, while response-level length normalization ties the magnitude of the update to the length of the output. We propose Hysteretic Policy Optimization (HPO), a minimal modification of GRPO that reduces the weight of negative-advantage updates and replaces per-response length normalization with mean-length normalization. We further introduce Adaptive HPO (A-HPO), which sets the hysteretic weight based on batch-level advantage-sign statistics, thereby removing the need for tuning a fixed hysteretic weight. In our TeleLogs and Countdown experiments, A-HPO improves the reward per update compared to GRPO, with the largest gains in early sparse reward regimes. On TeleLogs, A-HPO achieves a final reward of 0.84, outperforming SAPO by 5%, GSPO by 11%, and GRPO by 15%, while maintaining a comparable response-length. On Countdown, A-HPO achieves the largest gains in initial and most difficult configurations across 1.5B-7B models. Ablation studies on the hysteretic weight show that the gains of A-HPO come from better balancing the contributions of positive and negative advantages compared to positive-only or fully symmetric updates.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 11 of its 14 references that are in the map. First posted on arXiv 28 May 2026.

Builds on

11 of 14 references.

References · 11

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  4. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
  5. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  6. 2026Reinforcement Learning via Self-Distillationcited by 285
  7. 2025A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforcecited by 141
  8. 2025Rewarding the Unlikely: Lifting GRPO Beyond Distribution Sharpeningcited by 78
  9. 2025RewardMap: Tackling Sparse Rewards in Fine-grained Visual Reasoning via Multi-Stage Reinforcement Learningcited by 20
  10. 2026DenseGRPO: From Sparse to Dense Reward for Flow Matching Model Alignmentcited by 20
  11. 2025Reasoning Language Models for Root Cause Analysis in 5G Wireless Networkscited by 12

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization
  2. 2026GOPO: Policy Optimization using Ranked Rewards
  3. 2025Advantage Shaping as Surrogate Reward Maximization: Unifying Pass@K Policy Gradients
  4. 2025GHPO: Adaptive Guidance for Stable and Efficient LLM Reinforcement Learning
  5. 2025GRPOformer: Advancing Hyperparameter Optimization via Group Relative Policy Optimization
  6. 2026How Off-Policy Can GRPO Be? Mu-GRPO for Efficient LLM Reinforcement Learning
  7. 2025Reinforcement Learning with Verifiable Rewards: GRPO's Effective Loss, Dynamics, and Success Amplification
  8. 2025BNPO: Beta Normalization Policy Optimization