mlsanityData to 7 Oct 2026

MHPO: Modulated Hazard-aware Policy Optimization for Stable Reinforcement Learning

arXiv 2603.16929 ↗First posted 14 Mar 2026cs.LG cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Regulating the importance ratio is critical for the training stability of Group Relative Policy Optimization (GRPO) based frameworks. However, prevailing ratio control methods, such as hard clipping, suffer from non-differentiable boundaries and vanishing gradient regions, failing to maintain gradient fidelity. Furthermore, these methods lack a hazard-aware mechanism to adaptively suppress extreme deviations, leaving the optimization process vulnerable to abrupt policy shifts. To address these challenges, we propose Modulated Hazard-aware Policy Optimization (MHPO), a novel framework designed for robust and stable reinforcement learning. The proposed MHPO introduces a Log-Fidelity Modulator (LFM) to map unbounded importance ratios into a bounded, differentiable domain. This mechanism effectively prevents high-variance outlier tokens from destabilizing the loss landscape while ensuring global gradient stability. Complementarily, a Decoupled Hazard Penalty (DHP) integrates cumulative hazard functions from survival analysis to independently regulate positive and negative policy shifts. By shaping the optimization landscape with hazard-aware penalties, the proposed MHPO achieves fine-grained regulation of asymmetric policy shifts simultaneously mitigating mode collapse from over-expansion and preventing policy erosion from catastrophic contraction within a stabilized trust region. Extensive evaluations on diverse reasoning benchmarks across both text-based and vision-language tasks demonstrate that MHPO consistently outperforms existing methods, achieving superior performance while significantly enhancing training stability.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 48 of its 57 references that are in the map. First posted on arXiv 14 Mar 2026.

Builds on

48 of 57 references.

References · 48

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2025Qwen3 Technical Reportcited by 8,195
  5. 2025Qwen2.5-VL Technical Reportcited by 5,418
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  7. 2024Qwen2.5 Technical Reportcited by 4,713
  8. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  9. 2015Trust Region Policy Optimizationcited by 2,503
  10. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  11. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  12. 2025Group Sequence Policy Optimizationcited by 788
36 more
  1. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  2. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 573
  3. 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Datasetcited by 516
  4. 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Modelcited by 515
  5. 2025Visual-RFT: Visual Reinforcement Fine-Tuningcited by 501
  6. 2021Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoningcited by 376
  7. 2025R1-Onevision: Advancing Generalized Multimodal Reasoning through Cross-Modal Formalizationcited by 364
  8. 2023Secrets of RLHF in Large Language Models Part I: PPOcited by 226
  9. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraintscited by 174
  10. 2025GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoningcited by 147
  11. 2022RL with KL penalties is better viewed as Bayesian inferencecited by 130
  12. 2025What's Behind PPO's Collapse in Long-CoT? Value Optimization Holds the Secretcited by 95
  13. 2025Soft Adaptive Policy Optimizationcited by 89
  14. 2025SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy Optimizationcited by 87
  15. 2025NoisyRollout: Reinforcing Visual Reasoning with Data Augmentationcited by 86
  16. 2025GRPO-LEAD: A Difficulty-Aware Reinforcement Learning Approach for Concise Mathematical Reasoning in Language Modelscited by 81
  17. 2025Vision-R1: Evolving Human-Free Alignment in Large Vision-Language Models via Vision-Guided Reinforcement Learningcited by 68
  18. 2025Tapered Off-Policy REINFORCE: Stable and efficient reinforcement learning for LLMscited by 61
  19. 2025R1-ShareVL: Incentivizing Reasoning Capability of Multimodal Large Language Models via Share-GRPOcited by 52
  20. 2025VisionReasoner: Unified Reasoning-Integrated Visual Perception via Reinforcement Learningcited by 50
  21. 2025GRPO-CARE: Consistency-Aware Reinforcement Learning for Multimodal Reasoningcited by 46
  22. 2025SophiaVL-R1: Reinforcing MLLMs Reasoning with Thinking Rewardcited by 45
  23. 2024Tuning Large Multimodal Models for Videos using Reinforcement Learning from AI Feedbackcited by 36
  24. 2025Ignore the KL Penalty! Boosting Exploration on Critical Tokens to Enhance RL Fine-Tuningcited by 33
  25. 2025Boosting MLLM Reasoning with Text-Debiased Hint-GRPOcited by 32
  26. 2025NGRPO: Negative-enhanced Group Relative Policy Optimizationcited by 31
  27. 2025On the Design of KL-Regularized Policy Gradient Algorithms for LLM Reasoningcited by 30
  28. 2025Asymmetric REINFORCE for off-Policy Reinforcement Learning: Balancing positive and negative rewardscited by 29
  29. 2025On a few pitfalls in KL divergence gradient estimation for RLcited by 27
  30. 2025When Importance Sampling Misallocates Credit: Asymmetric Ratios for Outcome-Supervised RLcited by 21
  31. 2024Catastrophic Goodhart: regularizing RLHF with KL divergence does not mitigate heavy-tailed reward misspecificationcited by 20
  32. 2025Rethinking Sample Polarity in Reinforcement Learning with Verifiable Rewardscited by 19
  33. 2025Stepwise Guided Policy Optimization: Coloring your Incorrect Reasoning in GRPOcited by 18
  34. 2025Kalman Filter Enhanced GRPO for Reinforcement Learning-Based Language Model Reasoningcited by 14
  35. 2025Revisiting Visual Understanding in Multimodal Reasoning through a Lens of Image Perturbationcited by 14
  36. 2025Multi-Layer GRPO: Enhancing Reasoning and Self-Correction in Large Language Modelscited by 9

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026SAFE: Stable Alignment Finetuning with Entropy-Aware Predictive Control for Reinforcement Learning from Human Feedback (RLHF)
  2. 2026Ratio-Variance Regularized Policy Optimization
  3. 2026Safe RLHF Beyond Expectation: Stochastic Dominance for Universal Spectral Risk Control
  4. 2026Bounded Ratio Reinforcement Learning
  5. 2026ANO: Robust Policy Optimization via Bounded, Redescending Gain Fields
  6. 2024FAWAC: Feasibility Informed Advantage Weighted Regression for Persistent Safety in Offline Reinforcement Learning
  7. 2025Exchange Policy Optimization Algorithm for Semi-Infinite Safe Reinforcement Learning
  8. 2025Entropy Ratio Clipping as a Soft Global Constraint for Stable Reinforcement Learning