mlsanityData to 7 Oct 2026

Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs

arXiv 2406.10216 ↗First posted 14 Jun 2024cs.CL cs.AIcited by 142

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reward models trained on human preference data have been proven to effectively align Large Language Models (LLMs) with human intent within the framework of reinforcement learning from human feedback (RLHF). However, current reward models have limited generalization capabilities to unseen prompts and responses, which can lead to an unexpected phenomenon known as reward over-optimization, resulting in a decline in actual performance due to excessive optimization of rewards. While previous research has advocated for constraining policy optimization, our study introduces a novel approach to enhance the reward model's generalization ability against distribution shifts by regularizing the hidden states. Specifically, we retain the base model's language model head and incorporate a suite of text-generation losses to preserve the hidden states' text-generation capabilities, while concurrently learning a reward head behind the same hidden states. Our experimental results demonstrate that the introduced regularization technique markedly improves the accuracy of learned reward models across a variety of out-of-distribution (OOD) tasks and effectively alleviates the over-optimization issue in RLHF, offering a more reliable and robust preference learning paradigm.

Key facts. Cited by 142 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 59 of its 75 references that are in the map. First posted on arXiv 14 Jun 2024.

Builds on

59 of 75 references.

Closest ancestors · 24

Same direction, newest first

  1. 2024Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF29 May
  2. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer26 May
  3. 2024Inverse-RLignment: Large Language Model Alignment from Demonstrations through Inverse Reinforcement Learning24 May
  4. 2024RLHF Workflow: From Reward Modeling to Online RLHF13 May
  5. 2024MetaRM: Shifted Distributions Alignment via Meta-Learning1 May
  6. 2024DPO Meets PPO: Reinforced Token Optimization for RLHF29 Apr
  7. 2024RewardBench: Evaluating Reward Models for Language Modeling20 Mar
  8. 2024Overcoming Reward Overoptimization via Adversarial Policy Optimization with Lightweight Uncertainty Estimation8 Mar
  9. 2024Bayesian Reward Models for LLM Alignment20 Feb
  10. 2024Corruption Robust Offline Reinforcement Learning with Human Feedback9 Feb
  11. 2024Dense Reward for Free in Reinforcement Learning from Human Feedback1 Feb
  12. 2024Improving Reinforcement Learning from Human Feedback with Efficient Reward Model Ensemble30 Jan
12 more
  1. 2024Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHF29 Jan
  2. 2024WARM: On the Benefits of Weight Averaged Reward Models22 Jan
  3. 2024Secrets of RLHF in Large Language Models Part II: Reward Modeling11 Jan
  4. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking14 Dec
  5. 2023Confronting Reward Model Overoptimization with Constrained RLHF6 Oct
  6. 2023Reward Model Ensembles Help Mitigate Overoptimization4 Oct
  7. 2023Reinforced Self-Training (ReST) for Language Modeling17 Aug
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  9. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr
  10. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  11. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  12. 2019Fine-Tuning Language Models from Human Preferences18 Sep

Other references · 35

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  8. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  9. 2020Scaling Laws for Neural Language Modelscited by 6,411
  10. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  11. 2023Mistral 7Bcited by 3,065
  12. 2020Learning to summarize from human feedbackcited by 2,562
23 more
  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  2. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  3. 2016Concrete Problems in AI Safetycited by 1,582
  4. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  5. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  6. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
  7. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  8. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  9. 2023Instruction Tuning with GPT-4cited by 676
  10. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  11. 2020Is Pessimism Provably Efficient for Offline RL?cited by 362
  12. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustmentcited by 171
  13. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewardscited by 145
  14. 2022Rethinking Goal-conditioned Supervised Learning and Its Connection to Offline RLcited by 89
  15. 2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothingcited by 83
  16. 2023Query-Dependent Prompt Evaluation and Optimization with Offline Inverse RLcited by 37
  17. 2023What is Essential for Unseen Goal Generalization of Offline Goal-conditioned RL?cited by 36
  18. 2023Corruption-Robust Offline Reinforcement Learning with General Function Approximationcited by 30
  19. 2023Adversarial Preference Optimization: Enhancing Your Alignment via RM-LLM Gamecited by 27
  20. 2023Spurious Feature Diversification Improves Out-of-distribution Generalizationcited by 26
  21. 2023Towards Robust Offline Reinforcement Learning under Diverse Data Corruptioncited by 24
  22. 2024Confidence-aware Reward Optimization for Fine-tuning Text-to-Image Modelscited by 9
  23. 2024On the Benefits of Over-parameterization for Out-of-Distribution Generalizationcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 25; 2025: 69; 2026: 48.

142 papers.

Sort:

By area:

  1. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 465
  2. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 322
  3. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 294
  4. 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agentscited by 213
  5. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Stylecited by 168
  6. 2024DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical Reasoning Robustness of Vision Language Modelscited by 143
  7. 2024Critique-out-Loud Reward Modelscited by 105
  8. 2025GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoningcited by 94
  9. 2025Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignmentcited by 91
  10. 2025Reward Shaping to Mitigate Reward Hacking in RLHFcited by 88
  11. 2025InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Modelcited by 73
  12. 2024Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknowncited by 71
38 more
  1. 2024Reinforcement Learning Enhanced LLMs: A Surveycited by 69
  2. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  3. 2024Rethinking Bradley-Terry Models in Preference-Based Reward Modeling: Foundations, Theory, and Alternativescited by 48
  4. 2025Writing-Zero: Bridge the Gap Between Non-verifiable Tasks and Verifiable Rewardscited by 47
  5. 2025A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoningcited by 40
  6. 2024Direct Judgement Preference Optimizationcited by 33
  7. 2024Reward-Robust RLHF in LLMscited by 32
  8. 2025Inference-Time Reward Hacking in Large Language Modelscited by 32
  9. 2025Skywork-VL Reward: An Effective Reward Model for Multimodal Understanding and Reasoningcited by 30
  10. 2024Post-hoc Reward Calibration: A Case Study on Length Biascited by 29
  11. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applicationscited by 27
  12. 2024JuStRank: Benchmarking LLM Judges for System Rankingcited by 27
  13. 2024On the Limited Generalization Capability of the Implicit Reward Model Induced by Direct Preference Optimizationcited by 26
  14. 2026Open Rubric System: Scaling Reinforcement Learning with Pairwise Adaptive Rubriccited by 26
  15. 2025GRAM: A Generative Foundation Reward Model for Reward Generalizationcited by 25
  16. 2024Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Surveycited by 23
  17. 2024Improving Context-Aware Preference Modeling for Language Modelscited by 22
  18. 2024RAG-RewardBench: Benchmarking Reward Models in Retrieval Augmented Generation for Preference Alignmentcited by 22
  19. 2025Reasoning as an Adaptive Defense for Safetycited by 22
  20. 2024Evaluating Robustness of Reward Models for Mathematical Reasoningcited by 21
  21. 2025Rejected Dialects: Biases Against African American Language in Reward Modelscited by 20
  22. 2024Towards a Unified View of Preference Learning for Large Language Models: A Surveycited by 17
  23. 2025Mitigating Reward Over-Optimization in RLHF via Behavior-Supported Regularizationcited by 14
  24. 2025On the Robustness of Reward Models for Language Model Alignmentcited by 13
  25. 2025Distributionally Robust Reinforcement Learning with Human Feedbackcited by 12
  26. 2026AgentV-RL: Scaling Reward Modeling with Agentic Verifiercited by 12
  27. 2024Test-Time Alignment via Hypothesis Reweightingcited by 11
  28. 2025Semantically-Aware Rewards for Open-Ended R1 Training in Free-Form Generationcited by 11
  29. 2025Structural Reward Model: Enhancing Interpretability, Efficiency, and Scalability in Reward Modelingcited by 11
  30. 2025Rethinking Diverse Human Preference Learning through Principal Component Analysiscited by 10
  31. 2025Rethinking Reward Model Evaluation Through the Lens of Reward Overoptimizationcited by 10
  32. 2025Reward Model Overoptimisation in Iterated RLHFcited by 10
  33. 2025Frictional Agent Alignment Framework: Slow Down and Don't Break Thingscited by 10
  34. 2025Reward Model Interpretability via Optimal and Pessimal Tokenscited by 10
  35. 2025Bradley-Terry and Multi-Objective Reward Modeling Are Complementarycited by 10
  36. 2025Reward-Weighted Sampling: Enhancing Non-Autoregressive Characteristics in Masked Diffusion LLMscited by 10
  37. 2025MiCRo: Mixture Modeling and Context-aware Routing for Personalized Preference Learningcited by 9
  38. 2025Capturing Individual Human Preferences with Reward Featurescited by 8

50 most cited of 142 All 142 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024T-REG: Preference Optimization with Token-Level Reward Regularization
  2. 2024Fine-Tuning Language Models with Reward Learning on Policy
  3. 2025Generalist Reward Models: Found Inside Large Language Models
  4. 2024On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularization
  5. 2026Gradient Regularization Mitigates Reward Hacking in Reinforcement Learning from Human Feedback and Verifiable Rewards
  6. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  7. 2025Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignment
  8. 2025Off-Policy Corrected Reward Modeling for Reinforcement Learning from Human Feedback