mlsanityData to 7 Oct 2026

Confronting Reward Model Overoptimization with Constrained RLHF

arXiv 2310.04373 ↗First posted 6 Oct 2023cs.LG cs.AIcited by 96

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Large language models are typically aligned with human preferences by optimizing textitreward models (RMs) fitted to human feedback. However, human preferences are multi-faceted, and it is increasingly common to derive reward from a composition of simpler reward models which each capture a different aspect of language quality. This itself presents a challenge, as it is difficult to appropriately weight these component RMs when combining them. Compounding this difficulty, because any RM is only a proxy for human evaluation, this process is vulnerable to textitoveroptimization, wherein past a certain point, accumulating higher reward is associated with worse human ratings. In this paper, we perform, to our knowledge, the first study on overoptimization in composite RMs, showing that correlation between component RMs has a significant effect on the locations of these points. We then introduce an approach to solve this issue using constrained reinforcement learning as a means of preventing the agent from exceeding each RM's threshold of usefulness. Our method addresses the problem of weighting component RMs by learning dynamic weights, naturally expressed by Lagrange multipliers. As a result, each RM stays within the range at which it is an effective proxy, improving evaluation performance. Finally, we introduce an adaptive method using gradient-free optimization to identify and optimize towards these points during a single run.

Key facts. Cited by 96 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 57 of its 88 references that are in the map. First posted on arXiv 6 Oct 2023.

Builds on

57 of 88 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback27 Jul
  2. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Training2 Jun
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  4. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tears11 Apr
  5. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  6. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  7. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 50

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  7. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  8. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  9. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  10. 2020Learning to summarize from human feedbackcited by 2,562
  11. 2015Trust Region Policy Optimizationcited by 2,503
  12. 2020Extracting Training Data from Large Language Modelscited by 2,144
38 more
  1. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  2. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  3. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  4. 2017Constrained Policy Optimizationcited by 928
  5. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
  6. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  7. 2017DailyDialog: A Manually Labelled Multi-turn Dialogue Datasetcited by 649
  8. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  9. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  10. 2018Reward Constrained Policy Optimizationcited by 397
  11. 2021Unified Pre-training for Program Understanding and Generationcited by 361
  12. 2018Maximum a Posteriori Policy Optimisationcited by 318
  13. 2020Responsive Safety in Reinforcement Learning by PID Lagrangian Methodscited by 302
  14. 2018A Lyapunov-based Approach to Safe Reinforcement Learningcited by 278
  15. 2020Towards Continual Reinforcement Learning: A Review and Perspectivescited by 268
  16. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  17. 2017Distral: Robust Multitask Reinforcement Learningcited by 231
  18. 2022Fine-tuning language models to find agreement among humans with diverse preferencescited by 189
  19. 2020Exploration-Exploitation in Constrained MDPscited by 146
  20. 2019Constrained Reinforcement Learning Has Zero Duality Gapcited by 144
  21. 2020Provably Efficient Safe Exploration via Primal-Dual Policy Optimizationcited by 131
  22. 2020A Distributional Approach to Controlled Text Generationcited by 117
  23. 2020Reinforcement Learning for Non-Stationary Markov Decision Processes: The Blessing of (More) Optimismcited by 75
  24. 2018ROUGE 2.0: Updated and Improved Measures for Evaluation of Summarization Taskscited by 66
  25. 2019Information asymmetry in KL-regularized RLcited by 53
  26. 2020A Distributional View on Multi-Objective Policy Optimizationcited by 51
  27. 2020Deep Reinforcement Learning amidst Lifelong Non-Stationaritycited by 42
  28. 2019Non-Stationary Markov Decision Processes, a Worst-Case Approach using Model-Based Reinforcement Learning, Extended versioncited by 36
  29. 2022Discovering Policies with DOMiNO: Diversity Optimization Maintaining Near Optimalitycited by 35
  30. 2019Learning to Score Behaviors for Guided Policy Optimizationcited by 34
  31. 2023Benchmarking Large Language Model Capabilities for Conditional Generationcited by 29
  32. 2020Efficient Wasserstein Natural Gradients for Reinforcement Learningcited by 23
  33. 2023ReLOAD: Reinforcement Learning with Optimistic Ascent-Descent for Last-Iterate Convergence in Constrained MDPscited by 19
  34. 2020Balancing Constraints and Rewards with Meta-Gradient D4PGcited by 18
  35. 2021A First-Occupancy Representation for Reinforcement Learningcited by 14
  36. 2023Efficient Exploration via Epistemic-Risk-Seeking Policy Optimizationcited by 10
  37. 2021Towards an Understanding of Default Policies in Multitask Policy Optimizationcited by 8
  38. 2023A State Representation for Diminishing Rewardscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 32; 2025: 34; 2026: 28.

96 papers.

Sort:

By area:

  1. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  2. 2023Nash Learning from Human Feedbackcited by 220
  3. 2024Disentangling Length from Quality in Direct Preference Optimizationcited by 194
  4. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMscited by 142
  5. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithmscited by 139
  6. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109
  7. 2025Reward Shaping to Mitigate Reward Hacking in RLHFcited by 88
  8. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimizationcited by 78
  9. 2024InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modelingcited by 76
  10. 2024Robust Preference Optimization through Reward Model Distillationcited by 67
  11. 2024When Search Engine Services meet Large Language Models: Visions and Challengescited by 59
  12. 2023Uncertainty-Penalized Reinforcement Learning from Human Feedback with Diverse Reward LoRA Ensemblescited by 55
38 more
  1. 2025GRPO-Guard: Mitigating Implicit Over-Optimization in Flow Matching via Regulated Clippingcited by 48
  2. 2024Beyond Preferences in AI Alignmentcited by 46
  3. 2025Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Trainingcited by 46
  4. 2024The Perfect Blend: Redefining RLHF with Mixture of Judgescited by 43
  5. 2025Beyond Reward Hacking: Causal Rewards for Large Language Model Alignmentcited by 42
  6. 2024InfAlign: Inference-aware language model alignmentcited by 34
  7. 2025Chasing Moving Targets with Online Self-Play Reinforcement Learning for Safer Language Modelscited by 33
  8. 2024Confronting Reward Overoptimization for Diffusion Models: A Perspective of Inductive and Primacy Biasescited by 29
  9. 2024Eliminating Biased Length Reliance of Direct Preference Optimization via Down-Sampled KL Divergencecited by 29
  10. 2024Post-hoc Reward Calibration: A Case Study on Length Biascited by 29
  11. 2025The LLM Already Knows: Estimating LLM-Perceived Question Difficulty via Hidden Representationscited by 26
  12. 2024Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedbackcited by 25
  13. 2024Transforming and Combining Rewards for Aligning Large Language Modelscited by 24
  14. 2024One-Shot Safety Alignment for Large Language Models via Optimal Dualizationcited by 24
  15. 2024A Survey on Human Preference Learning for Large Language Modelscited by 23
  16. 2025What's In My Human Feedback? Learning Interpretable Descriptions of Preference Datacited by 22
  17. 2024Language Models in Dialogue: Conversational Maxims for Human-AI Interactionscited by 21
  18. 2024Policy Filtration for RLHF to Mitigate Noise in Reward Modelscited by 21
  19. 2024Countering Reward Over-optimization in LLM with Demonstration-Guided Reinforcement Learningcited by 20
  20. 2024Towards Data-Centric RLHF: Simple Metrics for Preference Dataset Comparisoncited by 20
  21. 2025Sotopia-RL: Reward Design for Social Intelligencecited by 20
  22. 2025Shop-R1: Rewarding LLMs to Simulate Human Behavior in Online Shopping via Reinforcement Learningcited by 19
  23. 2025The Energy Loss Phenomenon in RLHF: A New Perspective on Mitigating Reward Hackingcited by 18
  24. 2026RAGEN-2: Reasoning Collapse in Agentic RLcited by 16
  25. 2025A Survey on Progress in LLM Alignment from the Perspective of Reward Designcited by 15
  26. 2025Mitigating Reward Over-Optimization in RLHF via Behavior-Supported Regularizationcited by 14
  27. 2025On the Robustness of Reward Models for Language Model Alignmentcited by 13
  28. 2024ALaRM: Align Language Models via Hierarchical Rewards Modelingcited by 11
  29. 2025Rethinking Reward Model Evaluation Through the Lens of Reward Overoptimizationcited by 10
  30. 2025Reward Model Overoptimisation in Iterated RLHFcited by 10
  31. 2025Bradley-Terry and Multi-Objective Reward Modeling Are Complementarycited by 10
  32. 2024Unlocking Decoding-time Controllability: Gradient-Free Multi-Objective Alignment with Contrastive Promptscited by 9
  33. 2024Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedbackcited by 9
  34. 2025Alignment of large language models with constrained learningcited by 8
  35. 2024Rethinking the Role of Proxy Rewards in Language Model Alignmentcited by 7
  36. 2025OrthAlign: Orthogonal Subspace Decomposition for Non-Interfering Multi-Objective Alignmentcited by 7
  37. 2024Cost-Effective Proxy Reward Model Construction with On-Policy and Active Learningcited by 5
  38. 2024Enhancing Safety in Reinforcement Learning with Human Feedback via Rectified Policy Optimizationcited by 5

50 most cited of 96 All 96 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Reward Model Ensembles Help Mitigate Overoptimization
  2. 2025Off-Policy Corrected Reward Modeling for Reinforcement Learning from Human Feedback
  3. 2024Secrets of RLHF in Large Language Models Part II: Reward Modeling
  4. 2026Real-Time Aligned Reward Model beyond Semantics
  5. 2025A Systematic Analysis of Base Model Choice for Reward Modeling
  6. 2024Scalable Ensembling For Mitigating Reward Overoptimisation
  7. 2025A Unified Pairwise Framework for RLHF: Bridging Generative Reward Modeling and Policy Optimization
  8. 2025Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignment