mlsanityData to 14 Oct 2026

The Alignment Ceiling: Objective Mismatch in Reinforcement Learning from Human Feedback

arXiv 2311.00168 ↗First posted 31 Oct 2023cs.LGcited by 42

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement learning from human feedback (RLHF) has emerged as a powerful technique to make large language models (LLMs) more capable in complex settings. RLHF proceeds as collecting human preference data, training a reward model on said data, and optimizing a base ML model with respect to said reward for extrinsic evaluation metrics (e.g. MMLU, GSM8k). RLHF relies on many assumptions about how the various pieces fit together, such as a reward model capturing human preferences and an RL optimizer extracting the right signal from a reward model. As the RLHF process involves many distinct design decisions, it is easy to assume that multiple processes are correlated and therefore numerically linked. This apparent correlation is often not true, where reward models are easily overoptimized or RL optimizers can reduce performance on tasks not modeled in the data. Notable manifestations of models trained with imperfect RLHF systems are those that are prone to refusing basic requests for safety reasons or appearing lazy in generations. As chat model evaluation becomes increasingly nuanced, the reliance on a perceived link between reward model training, RL scores, and downstream performance drives these issues, which we describe as an objective mismatch. In this paper, we illustrate the causes of this issue, reviewing relevant literature from model-based reinforcement learning, and argue for solutions. By solving objective mismatch in RLHF, the ML models of the future will be more precisely aligned to user instructions for both safety and helpfulness.

Key facts. Cited by 42 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 48 of its 59 references that are in the map. First posted on arXiv 31 Oct 2023.

Builds on

48 of 59 references.

Closest ancestors · 19

Same direction, newest first

  1. 2023The History and Risks of Reinforcement Learning and Human Feedback20 Oct
  2. 2023Reward-Augmented Decoding: Efficient Controlled Text Generation With a Unidirectional Reward Model14 Oct
  3. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversity10 Oct
  4. 2023A Long Way to Go: Investigating Length Correlations in RLHF5 Oct
  5. 2023Reward Model Ensembles Help Mitigate Overoptimization4 Oct
  6. 2023Learning Optimal Advantage from Preferences and Mistaking it for Reward3 Oct
  7. 2023Pairwise Proximal Policy Optimization: Harnessing Relative Feedback for LLM Alignment30 Sep
  8. 2023The Trickle-down Impact of Reward (In-)consistency on RLHF28 Sep
  9. 2023Stabilizing RLHF through Advantage Model and Selective Rehearsal18 Sep
  10. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback27 Jul
  11. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Training2 Jun
  12. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
7 more
  1. 2023Shattering the Agent-Environment Interface for Fine-Tuning Inclusive Language Models19 May
  2. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  4. 2021WebGPT: Browser-assisted question-answering with human feedback17 Dec
  5. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  6. 2018Scalable agent alignment via reward modeling: a research direction19 Nov
  7. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 29

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2020Learning to summarize from human feedbackcited by 2,588
  6. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  7. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  8. 2023Large Language Models as Optimizerscited by 864
  9. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 774
  10. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
  11. 2022Improving alignment of dialogue agents via targeted human judgementscited by 531
  12. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 328
17 more
  1. 2023Alphazero-like Tree-Search can Guide Large Language Model Decoding and Trainingcited by 295
  2. 2022Teaching language models to support answers with verified quotescited by 292
  3. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 291
  4. 2021Recursively Summarizing Books with Human Feedbackcited by 290
  5. 2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 138
  6. 2020Objective Mismatch in Model-based Reinforcement Learningcited by 96
  7. 2017Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedbackcited by 72
  8. 2023Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generationcited by 55
  9. 2023SALMON: Self-Alignment with Instructable Reward Modelscited by 55
  10. 2022Investigating Compounding Prediction Errors in Learned Dynamics Modelscited by 47
  11. 2019Calibrated Model-Based Deep Reinforcement Learningcited by 31
  12. 2022Continual Learning for Instruction Following from Realtime Feedbackcited by 22
  13. 2020Learning Accurate Long-term Dynamics for Model-based Reinforcement Learningcited by 21
  14. 2023Leftover Lunch: Advantage-based Offline Reinforcement Learning for Language Modelscited by 17
  15. 2022Choices, Risks, and Reward Reports: Charting Public Policy for Reinforcement Learning Systemscited by 12
  16. 2022Personalized Reward Learning with Interaction-Grounded Learning (IGL)cited by 9
  17. 2023A Unified View on Solving Objective Mismatch in Model-Based Reinforcement Learningcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 22; 2025: 10; 2026: 9.

42 papers.

Sort:

By area:

  1. 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 236
  2. 2024Disentangling Length from Quality in Direct Preference Optimizationcited by 196
  3. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 147
  4. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithmscited by 143
  5. 2023HelpSteer: Multi-attribute Helpfulness Dataset for SteerLMcited by 142
  6. 2024Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedbackcited by 100
  7. 2024Red-Teaming for Generative AI: Silver Bullet or Security Theater?cited by 83
  8. 2024RMB: Comprehensively Benchmarking Reward Models in LLM Alignmentcited by 61
  9. 2024NeMo-Aligner: Scalable Toolkit for Efficient Model Alignmentcited by 41
  10. 2025Evaluation of Best-of-N Sampling Strategies for Language Model Alignmentcited by 39
  11. 2024Style Outweighs Substance: Failure Modes of LLM Judges in Alignment Benchmarkingcited by 35
  12. 2024LMUnit: Fine-grained Evaluation with Natural Language Unit Testscited by 33
30 more
  1. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 33
  2. 2024Creativity Has Left the Chat: The Price of Debiasing Language Modelscited by 29
  3. 2024Post-hoc Reward Calibration: A Case Study on Length Biascited by 29
  4. 2025SynthesizeMe! Inducing Persona-Guided Prompts for Personalized Reward Models in LLMscited by 29
  5. 2024Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedbackcited by 25
  6. 2024Regularized Best-of-N Sampling with Minimum Bayes Risk Objective for Language Model Alignmentcited by 25
  7. 2025AI-Slop to AI-Polish? Aligning Language Models through Edit-Based Writing Rewards and Test-time Computationcited by 25
  8. 2025Shrinking the Generation-Verification Gap with Weak Verifierscited by 24
  9. 2024Catastrophic Goodhart: regularizing RLHF with KL divergence does not mitigate heavy-tailed reward misspecificationcited by 23
  10. 2024Towards Data-Centric RLHF: Simple Metrics for Preference Dataset Comparisoncited by 21
  11. 2024Sail into the Headwind: Alignment via Robust Rewards and Dynamic Labels against Reward Hackingcited by 13
  12. 2025Scaling Evaluation-time Compute with Reasoning Models as Evaluatorscited by 13
  13. 2025Strategic Tradeoffs Between Humans and AI in Multi-Agent Bargainingcited by 11
  14. 2024Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedbackcited by 9
  15. 2024Drowning in Documents: Consequences of Scaling Reranker Inferencecited by 8
  16. 2025Bias Fitting to Mitigate Length Bias of Reward Model in RLHFcited by 7
  17. 2025Disentangling Length Bias In Preference Learning Via Response-Conditioned Modelingcited by 6
  18. 2024It Takes Two: On the Seamlessness between Reward and Policy Model in RLHFcited by 5
  19. 2025Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPOcited by 5
  20. 2024SEAL: Systematic Error Analysis for Value ALignmentcited by 4
  21. 2024Enhancing Reinforcement Learning with Label-Sensitive Reward for Natural Language Understandingcited by 2
  22. 2026Mining or Synthesis? Rethinking Exploration Efficiency in Iterative Alignment of Mathematical Reasoningcited by 2
  23. 2025Establishing Reliability Metrics for Reward Models in Large Language Modelscited by 1
  24. 2026IRPM: Intergroup Relative Preference Modeling for Pointwise Generative Reward Modelscited by 1
  25. 2026MOSAIC: Composable Safety Alignment with Modular Control Tokenscited by 1
  26. 2026Discovering Implicit Large Language Model Alignment Objectivescited by 0
  27. 2026Robust Post-Training for Generative Recommenders: Why Exponential Reward-Weighted SFT Outperforms RLHFcited by 0
  28. 2026In-Context Reward Adaptation for Robust Preference Modelingcited by 0
  29. 2026What Do People Actually Want From AI? Mapping Preference Pluralitycited by 0
  30. 2026RLVR2: Reinforcement Learning with Verifiable Rubric-based Rankingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Secrets of RLHF in Large Language Models Part II: Reward Modeling
  2. 2024Reward-Robust RLHF in LLMs
  3. 2023Secrets of RLHF in Large Language Models Part I: PPO
  4. 2024Does RLHF Scale? Exploring the Impacts From Data, Model, and Method
  5. 2024Reward Difference Optimization For Sample Reweighting In Offline RLHF
  6. 2023A Survey of Reinforcement Learning from Human Feedback
  7. 2025Robust Reinforcement Learning from Human Feedback for Large Language Models Fine-Tuning
  8. 2025Aligning to What? Limits to RLHF Based Alignment