mlsanityData to 14 Oct 2026

Mitigating Preference Hacking in Policy Optimization with Pessimism

arXiv 2503.06810 ↗First posted 10 Mar 2025cs.LG cs.AIcited by 8

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

This work tackles the problem of overoptimization in reinforcement learning from human feedback (RLHF), a prevalent technique for aligning models with human preferences. RLHF relies on reward or preference models trained on fixed preference datasets, and these models are unreliable when evaluated outside the support of this preference data, leading to the common reward or preference hacking phenomenon. We propose novel, pessimistic objectives for RLHF which are provably robust to overoptimization through the use of pessimism in the face of uncertainty, and design practical algorithms, P3O and PRPO, to optimize these objectives. Our approach is derived for the general preference optimization setting, but can be used with reward models as well. We evaluate P3O and PRPO on the tasks of fine-tuning language models for document summarization and creating helpful assistants, demonstrating remarkable resilience to overoptimization.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 39 of its 58 references that are in the map. First posted on arXiv 10 Mar 2025.

Builds on

39 of 58 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF29 May
  2. 2024Online Iterative Reinforcement Learning from Human Feedback with General Preference Model11 Feb
  3. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedback8 Jan
  4. 2023A Long Way to Go: Investigating Length Correlations in RLHF5 Oct
  5. 2023Reward Model Ensembles Help Mitigate Overoptimization4 Oct
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  7. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  9. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 30

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2020Learning to summarize from human feedbackcited by 2,588
  4. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  5. 2016Concrete Problems in AI Safetycited by 1,642
  6. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  7. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  8. 2023PaLM 2 Technical Reportcited by 1,162
  9. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  10. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  11. 2019Behavior Regularized Offline Reinforcement Learningcited by 662
  12. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 590
18 more
  1. 2020Is Pessimism Provably Efficient for Offline RL?cited by 364
  2. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 287
  3. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 284
  4. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 269
  5. 2024Direct Language Model Alignment from Online AI Feedbackcited by 260
  6. 2024Disentangling Length from Quality in Direct Preference Optimizationcited by 196
  7. 2022Scaling Up Models and Data with textttt5x and textttseqiocited by 187
  8. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignmentcited by 157
  9. 2017Safe Policy Improvement with Baseline Bootstrappingcited by 144
  10. 2020What are the Statistical Limits of Offline RL with Linear Function Approximation?cited by 136
  11. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learningcited by 110
  12. 2020The Importance of Pessimism in Fixed-Dataset Policy Optimizationcited by 99
  13. 2024Human Alignment of Large Language Models through Online Preference Optimisationcited by 91
  14. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimizationcited by 78
  15. 2024Robust Preference Optimization through Reward Model Distillationcited by 67
  16. 2023The In-Sample Softmax for Offline Reinforcement Learningcited by 33
  17. 2022When is Offline Two-Player Zero-Sum Markov Game Solvable?cited by 28
  18. 2023Offline Learning in Markov Games with General Function Approximationcited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 6.

8 papers.

  1. 2025Inference-Time Reward Hacking in Large Language Modelscited by 34
  2. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 33
  3. 2025Learning a Pessimistic Reward Model in RLHFcited by 5
  4. 2026LibraGen: Playing a Balance Game in Subject-Driven Video Generationcited by 2
  5. 2026Theoretical Foundations and Effective Algorithms for Policy-Aware Simulator Learningcited by 2
  6. 2026A Unifying Lens on Reward Uncertainty in RLHFcited by 1
  7. 2026Back to Blackwell: Closing the Loop on Intransitivity in Multi-Objective Preference Fine-Tuningcited by 0
  8. 2026Can You Trust Your Preference Models? Stackelberg Games for Alignmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  2. 2026A Regret Minimization Framework on Preference Learning in Large Language Models
  3. 2025Greedy Sampling Is Provably Efficient for RLHF
  4. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint
  5. 2025PILAF: Optimal Human Preference Sampling for Reward Modeling
  6. 2025Towards Efficient Online Exploration for Reinforcement Learning with Human Feedback
  7. 2023Contrastive Preference Learning: Learning from Human Feedback without RL
  8. 2025Direct Preference Optimization with Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences