mlsanityData to 14 Oct 2026

Off-Policy Corrected Reward Modeling for Reinforcement Learning from Human Feedback

arXiv 2507.15507 ↗First posted 21 Jul 2025cs.LG cs.AI cs.CLcited by 6

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

also in 1 more
Abstract (arXiv)

Reinforcement Learning from Human Feedback (RLHF) allows us to train models, such as language models (LMs), to follow complex human preferences. In RLHF for LMs, we first train an LM using supervised fine-tuning, sample pairs of responses, obtain human feedback, and use the resulting data to train a reward model (RM). RL methods are then used to train the LM to maximize the reward given by the RM. As training progresses, the responses generated by the LM no longer resemble the responses seen by the RM during training, leading to the RM becoming inaccurate. The score given by the RM keeps increasing, but the learned behavior no longer matches the human preferences. This issue is known as overoptimization. We investigate overoptimization from the point of view of distribution shift and show that the shift results in an inconsistent estimate of the RM parameters, leading to an inconsistent estimate of the policy gradient. We propose Off-Policy Corrected Reward Modeling (OCRM), which iteratively off-policy corrects the RM using importance weighting, without requiring new labels or samples. This results in a more accurate RM, which empirically leads to an improved final policy. We validate our approach in experiments with summarization and chatbot datasets and show that it performs significantly better than standard RLHF methods and baselines. Our implementation is available at https://github.com/JohannesAck/OffPolicyCorrectedRewardModeling

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 24 of its 38 references that are in the map. First posted on arXiv 21 Jul 2025.

Builds on

24 of 38 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs24 Oct
  2. 2024The Perils of Optimizing Learned Reward Functions: Low Training Error Does Not Guarantee Low Regret22 Jun
  3. 2024RLHF Workflow: From Reward Modeling to Online RLHF13 May
  4. 2024Fine-Tuning Language Models with Reward Learning on Policy28 Mar
  5. 2024The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarization24 Mar
  6. 2024Improving Reinforcement Learning from Human Feedback with Efficient Reward Model Ensemble30 Jan
  7. 2024WARM: On the Benefits of Weight Averaged Reward Models22 Jan
  8. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking14 Dec
  9. 2023Reward Model Ensembles Help Mitigate Overoptimization4 Oct
  10. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  11. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  12. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 12

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2020Learning to summarize from human feedbackcited by 2,588
  4. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  5. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  6. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  7. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 819
  8. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  9. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
  10. 2022The Primacy Bias in Deep Reinforcement Learningcited by 247
  11. 2024The Importance of Online Data: Understanding Preference Fine-tuning via Coveragecited by 55
  12. 2024WPO: Enhancing RLHF with Weighted Preference Optimizationcited by 54

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026Gradient Regularization Mitigates Reward Hacking in Reinforcement Learning from Human Feedback and Verifiable Rewardscited by 8
  2. 2026Explaining and Preventing Alignment Collapse in Iterative RLHFcited by 4
  3. 2026CAFE: Self-Improving Search Agents Need Co-Evolving Feedbackcited by 2
  4. 2026Mitigating Reward Hacking in RLHF via Advantage Sign Robustnesscited by 1
  5. 2026Wasserstein Distributionally Robust Regret Optimization for Reinforcement Learning from Human Feedbackcited by 0
  6. 2026CoRe: Co-Evolving Reward Models for Mitigating Latent Reward Hacking in Video Diffusion Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Reward Model Overoptimisation in Iterated RLHF
  2. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  3. 2026Real-Time Aligned Reward Model beyond Semantics
  4. 2025Mitigating Reward Over-Optimization in RLHF via Behavior-Supported Regularization
  5. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
  6. 2024Scalable Ensembling For Mitigating Reward Overoptimisation
  7. 2025Generalist Reward Models: Found Inside Large Language Models
  8. 2024Solving the Inverse Alignment Problem for Efficient RLHF