mlsanityData to 7 Oct 2026

Unifying Stable Optimization and Reference Regularization in RLHF

arXiv 2602.11523 ↗First posted 12 Feb 2026cs.LGcited by 4

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement Learning from Human Feedback (RLHF) has advanced alignment capabilities significantly but remains hindered by two core challenges: reward hacking and stable optimization. Current solutions independently address these issues through separate regularization strategies, specifically a KL-divergence penalty against a supervised fine-tuned model (π0) to mitigate reward hacking, and policy ratio clipping towards the current policy (πt) to promote stable alignment. However, the implicit trade-off arising from simultaneously regularizing towards both π0 and πt remains under-explored. In this paper, we introduce a unified regularization approach that explicitly balances the objectives of preventing reward hacking and maintaining stable policy updates. Our simple yet principled alignment objective yields a weighted supervised fine-tuning loss with a superior trade-off, which demonstrably improves both alignment results and implementation complexity. Extensive experiments across diverse benchmarks validate that our method consistently outperforms RLHF and online preference learning methods, achieving enhanced alignment performance and stability.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 64 of its 80 references that are in the map. First posted on arXiv 12 Feb 2026.

Builds on

64 of 80 references.

Closest ancestors · 17

Same direction, newest first

  1. 2024Sail into the Headwind: Alignment via Robust Rewards and Dynamic Labels against Reward Hacking12 Dec
  2. 2024HelpSteer2-Preference: Complementing Ratings with Preferences2 Oct
  3. 2024Catastrophic Goodhart: regularizing RLHF with KL divergence does not mitigate heavy-tailed reward misspecification19 Jul
  4. 2024WARP: On the Benefits of Weight Averaged Rewarded Policies24 Jun
  5. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs14 Jun
  6. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms5 Jun
  7. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer26 May
  8. 2024RLHF Workflow: From Reward Modeling to Online RLHF13 May
  9. 2023A Survey of Reinforcement Learning from Human Feedback22 Dec
  10. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint18 Dec
  11. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback1 Sep
  12. 2023Secrets of RLHF in Large Language Models Part I: PPO11 Jul
5 more
  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  2. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedback17 May
  3. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  5. 2019Fine-Tuning Language Models from Human Preferences18 Sep

Other references · 47

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2024The Llama 3 Herd of Modelscited by 14,192
  6. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  7. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  8. 2025Qwen3 Technical Reportcited by 8,195
  9. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  10. 2024Qwen2.5 Technical Reportcited by 4,713
  11. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  12. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
35 more
  1. 2020Learning to summarize from human feedbackcited by 2,562
  2. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  3. 2015Trust Region Policy Optimizationcited by 2,503
  4. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  5. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  6. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  7. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  8. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  9. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  10. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  11. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  12. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 556
  13. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translationcited by 390
  14. 2020Critic Regularized Regressioncited by 269
  15. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
  16. 2024Direct Language Model Alignment from Online AI Feedbackcited by 257
  17. 2024Language Models Learn to Mislead Humans via RLHFcited by 118
  18. 2024Learn Your Reference Model for Real Good Alignmentcited by 56
  19. 2023SALMON: Self-Alignment with Instructable Reward Modelscited by 54
  20. 2025Advancing Reasoning in Large Language Models: Promising Methods and Approachescited by 43
  21. 2025Exploring Criteria of Loss Reweighting to Enhance LLM Unlearningcited by 39
  22. 2024Online DPO: Online Direct Preference Optimization with Fast-Slow Chasingcited by 38
  23. 2024CREAM: Consistency Regularized Self-Rewarding Language Modelscited by 35
  24. 2025A Survey of Direct Preference Optimizationcited by 35
  25. 2025On the Design of KL-Regularized Policy Gradient Algorithms for LLM Reasoningcited by 30
  26. 2023Machine Vision Therapy: Multimodal Large Language Models Can Enhance Visual Robustness via Denoising In-Context Learningcited by 27
  27. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applicationscited by 27
  28. 2024Mixed Preference Optimization: Reinforcement Learning with Data Selection and Better Reference Modelcited by 25
  29. 2025SimPER: A Minimalist Approach to Preference Alignment without Hyperparameterscited by 25
  30. 2024Understanding Reference Policies in Direct Preference Optimizationcited by 22
  31. 2020Stable Policy Optimization via Off-Policy Divergence Regularizationcited by 16
  32. 2024Multi-Reference Preference Optimization for Large Language Modelscited by 10
  33. 2024BPO: Staying Close to the Behavior LLM Creates Better Online LLM Alignmentcited by 9
  34. 2025URPO: A Unified Reward & Policy Optimization Framework for Large Language Modelscited by 5
  35. 2024Aligning Large Language Models from Self-Reference AI Feedback with one General Principlecited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Towards Robust LLM Post-Training: Automatic Failure Management for Reinforcement Fine-Tuningcited by 2
  2. 2026Boosting Reinforcement Learning with Verifiable Rewards via Randomly Selected Few-Shot Guidancecited by 0
  3. 2026SR-OPSD: Self-Referenced On-Policy Self-Distillationcited by 0
  4. 2026Advantage Scale Calibration Imbalance in Group-Relative Optimization under Low-Variance Rewards: Diagnosis and Bounded Recoverycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025RLHF in an SFT Way: From Optimal Solution to Reward-Weighted Alignment
  2. 2024Solving the Inverse Alignment Problem for Efficient RLHF
  3. 2026Generalisation of RLHF under Reward Shift and Clipped KL Regularisation
  4. 2025ROCM: RLHF on consistency models
  5. 2026Explaining and Preventing Alignment Collapse in Iterative RLHF
  6. 2025Beyond RLHF: A Unified Theoretical Framework of Alignment
  7. 2026Gradient Regularization Mitigates Reward Hacking in Reinforcement Learning from Human Feedback and Verifiable Rewards
  8. 2025KL-Regularized RLHF with Multiple Reference Models: Exact Solutions and Sample Complexity