mlsanityData to 13 Oct 2026

Learning a Pessimistic Reward Model in RLHF

arXiv 2505.20556 ↗First posted 26 May 2025cs.LGcited by 5

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

also in 1 more
Abstract (arXiv)

This work proposes `PET', a novel pessimistic reward fine-tuning method, to learn a pessimistic reward model robust against reward hacking in offline reinforcement learning from human feedback (RLHF). Traditional reward modeling techniques in RLHF train an imperfect reward model, on which a KL regularization plays a pivotal role in mitigating reward hacking when optimizing a policy. Such an intuition-based method still suffers from reward hacking, and the policies with large KL divergence from the dataset distribution are excluded during learning. In contrast, we show that when optimizing a policy on a pessimistic reward model fine-tuned through PET, reward hacking can be prevented without relying on any regularization. We test our methods on the standard TL;DR summarization dataset. We find that one can learn a high-quality policy on our pessimistic reward without using any regularization. Such a policy has a high KL divergence from the dataset distribution while having high performance in practice. In summary, our work shows the feasibility of learning a pessimistic reward model against reward hacking. The agent can greedily search for the policy with a high pessimistic reward without suffering from reward hacking.

Key facts. Cited by 5 papers in the map (data through 13 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 43 of its 51 references that are in the map. First posted on arXiv 26 May 2025.

Builds on

43 of 51 references.

Closest ancestors · 22

Same direction, newest first

  1. 2025Mitigating Reward Over-Optimization in RLHF via Behavior-Supported Regularization23 Mar
  2. 2025Mitigating Preference Hacking in Policy Optimization with Pessimism10 Mar
  3. 2025Logarithmic Regret for Online KL-Regularized Reinforcement Learning11 Feb
  4. 2025Sample-Efficient Reinforcement Learning from Human Feedback via Information-Directed Sampling8 Feb
  5. 2024RRM: Robust Reward Model Training Mitigates Reward Hacking20 Sep
  6. 2024Boosting Reward Model with Preference-Conditional Multi-Aspect Synthetic Data Generation22 Jul
  7. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms5 Jun
  8. 2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHF31 May
  9. 2024Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF29 May
  10. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer26 May
  11. 2024RLHF Workflow: From Reward Modeling to Online RLHF13 May
  12. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences4 Apr
10 more
  1. 2024The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarization24 Mar
  2. 2024WARM: On the Benefits of Weight Averaged Reward Models22 Jan
  3. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint18 Dec
  4. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking14 Dec
  5. 2023Making RL with Preference-based Feedback Efficient via Randomization23 Oct
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  7. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  8. 2022Causal Confusion and Reward Misidentification in Preference-Based Reward Learning13 Apr
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  10. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 21

  1. 2022Training language models to follow instructions with human feedbackcited by 15,856
  2. 2017Proximal Policy Optimization Algorithmscited by 14,592
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,072
  4. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,011
  5. 2020Learning to summarize from human feedbackcited by 2,575
  6. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,889
  7. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,761
  8. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 322
  9. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignmentcited by 156
  10. 2025A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforcecited by 143
  11. 2024Theoretical guarantees on the best-of-n alignment policycited by 120
  12. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learningcited by 111
9 more
  1. 2025Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignmentcited by 93
  2. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimizationcited by 78
  3. 2024Robust Preference Optimization through Reward Model Distillationcited by 67
  4. 2024Self-Exploring Language Models: Active Preference Elicitation for Online Alignmentcited by 63
  5. 2023Provable Offline Preference-Based Reinforcement Learningcited by 55
  6. 2024RS-DPO: A Hybrid Rejection Sampling and Direct Preference Optimization Method for Alignment of Large Language Modelscited by 48
  7. 2024Towards Efficient Exact Optimization of Language Model Alignmentcited by 37
  8. 2023Adversarial Model for Offline Reinforcement Learningcited by 31
  9. 2024Self-Play with Adversarial Critic: Provable and Scalable Offline Alignment for Language Modelscited by 16

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2025Learning from Failures: Understanding LLM Alignment through Failure-Aware Inverse RLcited by 1
  2. 2026Golden Handcuffs make safer AI agentscited by 1
  3. 2025Circuit-Aware Reward Training: A Mechanistic Framework for Longtail Robustness in RLHFcited by 0
  4. 2026Wasserstein Distributionally Robust Regret Optimization for Reinforcement Learning from Human Feedbackcited by 0
  5. 2026Themis: Training Robust Multilingual Code Reward Models for Flexible Multi-Criteria Scoringcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026A Unifying Lens on Reward Uncertainty in RLHF
  2. 2024Catastrophic Goodhart: regularizing RLHF with KL divergence does not mitigate heavy-tailed reward misspecification
  3. 2025Information-Theoretic Reward Modeling for Stable RLHF: Detecting and Mitigating Reward Hacking
  4. 2026Gradient Regularization Mitigates Reward Hacking in Reinforcement Learning from Human Feedback and Verifiable Rewards
  5. 2026Uncertainty-Aware Reward Modeling for Stable RLHF
  6. 2024InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modeling
  7. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  8. 2026RMB: Reward Model Boosting Mitigates Reward Hacking