mlsanityData to 13 Oct 2026

Scaling Laws for Reward Model Overoptimization

arXiv 2210.10760 ↗First posted 19 Oct 2022cs.LG stat.MLcited by 1,077

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

In reinforcement learning from human feedback, it is common to optimize against a reward model trained to predict human preferences. Because the reward model is an imperfect proxy, optimizing its value too much can hinder ground truth performance, in accordance with Goodhart's law. This effect has been frequently observed, but not carefully measured due to the expense of collecting human preference data. In this work, we use a synthetic setup in which a fixed "gold-standard" reward model plays the role of humans, providing labels used to train a proxy reward model. We study how the gold reward model score changes as we optimize against the proxy reward model using either reinforcement learning or best-of-n sampling. We find that this relationship follows a different functional form depending on the method of optimization, and that in both cases its coefficients scale smoothly with the number of reward model parameters. We also study the effect on this relationship of the size of the reward model dataset, the number of reward model and policy parameters, and the coefficient of the KL penalty added to the reward in the reinforcement learning setup. We explore the implications of these empirical results for theoretical considerations in AI alignment.

Key facts. Cited by 1,077 papers in the map (data through 13 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 43 of its 59 references that are in the map. First posted on arXiv 19 Oct 2022.

Builds on

43 of 59 references.

Closest ancestors · 6

Same direction, newest first

  1. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  2. 2022Uncertainty Estimation for Language Reward Models14 Mar
  3. 2021WebGPT: Browser-assisted question-answering with human feedback17 Dec
  4. 2018Scalable agent alignment via reward modeling: a research direction19 Nov
  5. 2018Reward learning from human preferences and demonstrations in Atari15 Nov
  6. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 37

  1. 2020Language Models are Few-Shot Learnerscited by 32,320
  2. 2022Training language models to follow instructions with human feedbackcited by 15,856
  3. 2017Proximal Policy Optimization Algorithmscited by 14,592
  4. 2020Scaling Laws for Neural Language Modelscited by 6,494
  5. 2013Intriguing properties of neural networkscited by 5,137
  6. 2020Learning to summarize from human feedbackcited by 2,575
  7. 2022Improving alignment of dialogue agents via targeted human judgementscited by 529
  8. 2020Scaling Laws for Autoregressive Generative Modelingcited by 514
  9. 2019On Evaluating Adversarial Robustnesscited by 447
  10. 2021Explaining Neural Scaling Lawscited by 355
  11. 2018Quantifying Generalization in Reinforcement Learningcited by 345
  12. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 320
25 more
  1. 2018Adversarial Attack on Graph Structured Datacited by 301
  2. 2022Teaching language models to support answers with verified quotescited by 291
  3. 2021Scaling Laws for Transfercited by 268
  4. 2018Adversarial Attacks and Defences: A Surveycited by 259
  5. 2019Risks from Learned Optimization in Advanced Machine Learning Systemscited by 254
  6. 2022The Alignment Problem from a Deep Learning Perspectivecited by 252
  7. 2021Gradient-based Adversarial Attacks against Text Transformerscited by 234
  8. 2022Defining and Characterizing Reward Hackingcited by 198
  9. 2018A Study on Overfitting in Deep Reinforcement Learningcited by 171
  10. 2017Tactics of Adversarial Attack on Deep Reinforcement Learning Agentscited by 165
  11. 2016Improving the Robustness of Deep Neural Networks via Stability Trainingcited by 156
  12. 2014Towards Deep Neural Network Architectures Robust to Adversarial Examplescited by 146
  13. 2018Categorizing Variants of Goodhart's Lawcited by 142
  14. 2022RL with KL penalties is better viewed as Bayesian inferencecited by 132
  15. 2018Generalization and Regularization in DQNcited by 117
  16. 2019Optimal Policies Tend to Seek Powercited by 117
  17. 2022Is Power-Seeking AI an Existential Risk?cited by 105
  18. 2019Observational Overfitting in Reinforcement Learningcited by 90
  19. 2018A Dissection of Overfitting and Generalization in Continuous Reinforcement Learningcited by 86
  20. 2018On Adversarial Examples for Character-Level Neural Machine Translationcited by 85
  21. 2021Consequences of Misaligned AIcited by 79
  22. 2018The Surprising Creativity of Digital Evolution: A Collection of Anecdotes from the Evolutionary Computation and Artificial Life Research Communitiescited by 78
  23. 2017Reinforcement Learning with a Corrupted Reward Channelcited by 68
  24. 2020A Neural Scaling Law from the Dimension of the Data Manifoldcited by 48
  25. 2017Occam's razor is insufficient to infer the preferences of irrational agentscited by 45

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2022: 3; 2023: 88; 2024: 242; 2025: 306; 2026: 438.

1,077 papers.

Sort:

By area:

  1. 2023Let's Verify Step by Stepcited by 3,672
  2. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,011
  3. 2023Towards Understanding Sycophancy in Language Modelscited by 1,018
  4. 2023Training Diffusion Models with Reinforcement Learningcited by 955
  5. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 954
  6. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 824
  7. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 751
  8. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 745
  9. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 697
  10. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 680
  11. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 648
  12. 2023ChemCrow: Augmenting large-language models with chemistry toolscited by 622
38 more
  1. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 558
  2. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  3. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 492
  4. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 488
  5. 2025Process Reinforcement through Implicit Rewardscited by 484
  6. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Modelscited by 473
  7. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 459
  8. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 454
  9. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 421
  10. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 390
  11. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 371
  12. 2025Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscationcited by 363
  13. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 347
  14. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 324
  15. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 322
  16. 2025MM-Eureka: Exploring the Frontiers of Multimodal Reasoning with Rule-based Reinforcement Learningcited by 299
  17. 2023Aligning Text-to-Image Diffusion Models with Reward Backpropagationcited by 286
  18. 2024Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Modelscited by 286
  19. 2023Preference Ranking Optimization for Human Alignmentcited by 284
  20. 2025Learning to Reason under Off-Policy Guidancecited by 272
  21. 2024Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoningcited by 271
  22. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 265
  23. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
  24. 2024Direct Language Model Alignment from Online AI Feedbackcited by 257
  25. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 245
  26. 2025Training Language Models to Reason Efficientlycited by 245
  27. 2023Large Language Model Alignment: A Surveycited by 243
  28. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  29. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 240
  30. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 229
  31. 2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 226
  32. 2023How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Compositioncited by 223
  33. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RLcited by 218
  34. 2023Inverse Scaling: When Bigger Isn't Bettercited by 207
  35. 2025Learning to Reason without External Rewardscited by 203
  36. 2024Aligning Modalities in Vision Large Language Models via Preference Fine-tuningcited by 197
  37. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hackingcited by 196
  38. 2024Disentangling Length from Quality in Direct Preference Optimizationcited by 195

50 most cited of 1,077 All 1,077 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Capturing Individual Human Preferences with Reward Features
  2. 2025Reward Models in Deep Reinforcement Learning: A Survey
  3. 2026When Errors Can Be Beneficial: A Categorization of Imperfect Rewards for Policy Gradient
  4. 2025Generalist Reward Models: Found Inside Large Language Models
  5. 2025From Demonstrations to Rewards: Alignment Without Explicit Human Preferences
  6. 2025Debiasing Reward Models by Representation Learning with Guarantees
  7. 2024Secrets of RLHF in Large Language Models Part II: Reward Modeling
  8. 2024Solving the Inverse Alignment Problem for Efficient RLHF