mlsanityData to 7 Oct 2026

Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms

arXiv 2406.02900 ↗First posted 5 Jun 2024cs.LG cs.AI cs.CLcited by 139

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement Learning from Human Feedback (RLHF) has been crucial to the recent success of Large Language Models (LLMs), however, it is often a complex and brittle process. In the classical RLHF framework, a reward model is first trained to represent human preferences, which is in turn used by an online reinforcement learning (RL) algorithm to optimize the LLM. A prominent issue with such methods is reward over-optimization or reward hacking, where performance as measured by the learned proxy reward model increases, but true quality plateaus or even deteriorates. Direct Alignment Algorithms (DDAs) like Direct Preference Optimization have emerged as alternatives to the classical RLHF pipeline by circumventing the reward modeling phase. However, although DAAs do not use a separate proxy reward model, they still commonly deteriorate from over-optimization. While the so-called reward hacking phenomenon is not well-defined for DAAs, we still uncover similar trends: at higher KL budgets, DAA algorithms exhibit similar degradation patterns to their classic RLHF counterparts. In particular, we find that DAA methods deteriorate not only across a wide range of KL budgets but also often before even a single epoch of the dataset is completed. Through extensive empirical experimentation, this work formulates and formalizes the reward over-optimization or hacking problem for DAAs and explores its consequences across objectives, training regimes, and model scales.

Key facts. Cited by 139 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 47 of its 68 references that are in the map. First posted on arXiv 5 Jun 2024.

Builds on

47 of 68 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024Countering Reward Over-optimization in LLM with Demonstration-Guided Reinforcement Learning30 Apr
  2. 2024Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHF29 Jan
  3. 2023Uncertainty-Penalized Reinforcement Learning from Human Feedback with Diverse Reward LoRA Ensembles30 Dec
  4. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking14 Dec
  5. 2023The Alignment Ceiling: Objective Mismatch in Reinforcement Learning from Human Feedback31 Oct
  6. 2023Confronting Reward Model Overoptimization with Constrained RLHF6 Oct
  7. 2023A Long Way to Go: Investigating Length Correlations in RLHF5 Oct
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  9. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedback17 May
  10. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  11. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  12. 2019Fine-Tuning Language Models from Human Preferences18 Sep
1 more
  1. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 34

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2013Intriguing properties of neural networkscited by 5,114
  8. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  9. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  10. 2016Concrete Problems in AI Safetycited by 1,582
  11. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  12. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
22 more
  1. 2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 767
  2. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  3. 2018Quantifying Generalization in Reinforcement Learningcited by 345
  4. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
  5. 2021Scaling Laws for Transfercited by 265
  6. 2017Inverse Reward Designcited by 246
  7. 2024Iterative Reasoning Preference Optimizationcited by 244
  8. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  9. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positivecited by 239
  10. 2024V-STaR: Training Verifiers for Self-Taught Reasonerscited by 214
  11. 2024Disentangling Length from Quality in Direct Preference Optimizationcited by 194
  12. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 184
  13. 2017Tactics of Adversarial Attack on Deep Reinforcement Learning Agentscited by 163
  14. 2018Categorizing Variants of Goodhart's Lawcited by 132
  15. 2018Generalization and Regularization in DQNcited by 117
  16. 2024Understanding the performance gap between online and offline alignment algorithmscited by 110
  17. 2018On Adversarial Examples for Character-Level Neural Machine Translationcited by 85
  18. 2017Reinforcement Learning with a Corrupted Reward Channelcited by 68
  19. 2023Dual RL: Unification and New Methods for Reinforcement and Imitation Learningcited by 48
  20. 2023Coherent Soft Imitation Learningcited by 21
  21. 2022A Ranking Game for Imitation Learningcited by 18
  22. 2024Understanding the Learning Dynamics of Alignment with Human Feedbackcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 32; 2025: 57; 2026: 50.

139 papers.

Sort:

By area:

  1. 2025Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscationcited by 338
  2. 2024Generative Reward Modelscited by 148
  3. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 129
  4. 2025Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedbackcited by 86
  5. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimizationcited by 78
  6. 2025T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scalingcited by 77
  7. 2024PERSONA: A Reproducible Testbed for Pluralistic Alignmentcited by 71
  8. 2024TangoFlux: Super Fast and Faithful Text to Audio Generation with Flow Matching and Clap-Ranked Preference Optimizationcited by 67
  9. 2024Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimizationcited by 66
  10. 2024WPO: Enhancing RLHF with Weighted Preference Optimizationcited by 53
  11. 2024EXAONE 3.5: Series of Large Language Models for Real-world Use Casescited by 50
  12. 2024Scaling Laws for Predicting Downstream Performance in LLMscited by 48
38 more
  1. 2024Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignmentcited by 46
  2. 2024The Perfect Blend: Redefining RLHF with Mixture of Judgescited by 43
  3. 2025Calibrated Multi-Preference Optimization for Aligning Diffusion Modelscited by 40
  4. 2025SafeDPO: A Simple Approach to Direct Preference Optimization with Enhanced Safetycited by 37
  5. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 30
  6. 2024Eliminating Biased Length Reliance of Direct Preference Optimization via Down-Sampled KL Divergencecited by 29
  7. 2024AlphaDPO: Adaptive Reward Margin for Direct Preference Optimizationcited by 27
  8. 2024Selective Preference Optimization via Token-Level Reward Function Estimationcited by 25
  9. 2024Process Supervision-Guided Policy Optimization for Code Generationcited by 25
  10. 2025Online Preference Alignment for Language Models via Count-based Explorationcited by 24
  11. 2024Does RLHF Scale? Exploring the Impacts From Data, Model, and Methodcited by 22
  12. 2025Robust Multi-Objective Preference Alignment with Online DPOcited by 21
  13. 2025Interleaved Reasoning for Large Language Models via Reinforcement Learningcited by 21
  14. 2025MO-GRPO: Mitigating Reward Hacking of Group Relative Policy Optimization on Multi-Objective Problemscited by 21
  15. 2025The Energy Loss Phenomenon in RLHF: A New Perspective on Mitigating Reward Hackingcited by 18
  16. 2024Grounding by Trying: LLMs with Reinforcement Learning-Enhanced Retrievalcited by 17
  17. 2024How to Leverage Demonstration Data in Alignment for Large Language Model? A Self-Imitation Learning Perspectivecited by 14
  18. 2025AlphaPO: Reward Shape Matters for LLM Alignmentcited by 14
  19. 2025Generative AI Act II: Test Time Scaling Drives Cognition Engineeringcited by 14
  20. 2024Sail into the Headwind: Alignment via Robust Rewards and Dynamic Labels against Reward Hackingcited by 13
  21. 2025On the Robustness of Reward Models for Language Model Alignmentcited by 13
  22. 2024Fast and Slow Generating: An Empirical Study on Large and Small Language Models Collaborative Decodingcited by 12
  23. 2024Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inferencecited by 12
  24. 2025Distributionally Robust Reinforcement Learning with Human Feedbackcited by 12
  25. 2025Aligning Multimodal LLM with Human Preference: A Surveycited by 12
  26. 2024Little Giants: Synthesizing High-Quality Embedding Data at Scalecited by 11
  27. 2025ComPO: Preference Alignment via Comparison Oraclescited by 11
  28. 2025Aligning Spoken Dialogue Models from User Interactionscited by 11
  29. 2025Reward-aware Preference Optimization: A Unified Mathematical Framework for Model Alignmentcited by 10
  30. 2025Rethinking Reward Model Evaluation Through the Lens of Reward Overoptimizationcited by 10
  31. 2025Reward Model Overoptimisation in Iterated RLHFcited by 10
  32. 2025ConfPO: Exploiting Policy Model Confidence for Critical Token Selection in Preference Optimizationcited by 10
  33. 2025Hybrid Reward Normalization for Process-supervised Non-verifiable Agentic Taskscited by 10
  34. 2025Preference Learning Unlocks LLMs' Psycho-Counseling Skillscited by 9
  35. 2024Generalists vs. Specialists: Evaluating LLMs on Highly-Constrained Biophysical Sequence Optimization Taskscited by 8
  36. 2024Drowning in Documents: Consequences of Scaling Reranker Inferencecited by 8
  37. 2025Debate Helps Weak-to-Strong Generalizationcited by 8
  38. 2025Two Heads Are Better Than One: Dual-Model Verbal Reflection at Inference-Timecited by 8

50 most cited of 139 All 139 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Solving the Inverse Alignment Problem for Efficient RLHF
  2. 2025AlignDistil: Token-Level Language Model Alignment as Adaptive Policy Distillation
  3. 2024Elephant in the Room: Unveiling the Impact of Reward Model Quality in Alignment
  4. 2025A Principled Loss Function for Direct Language Model Alignment
  5. 2024Towards Reliable Alignment: Uncertainty-aware RLHF
  6. 2026Real-Time Aligned Reward Model beyond Semantics
  7. 2025RLHF in an SFT Way: From Optimal Solution to Reward-Weighted Alignment
  8. 2025A Survey on Progress in LLM Alignment from the Perspective of Reward Design