mlsanityData to 7 Oct 2026

Scalable agent alignment via reward modeling: a research direction

arXiv 1811.07871 ↗First posted 19 Nov 2018cs.LG cs.AI cs.NEcited by 438

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

One obstacle to applying reinforcement learning algorithms to real-world problems is the lack of suitable reward functions. Designing such reward functions is difficult in part because the user only has an implicit understanding of the task objective. This gives rise to the agent alignment problem: how do we create agents that behave in accordance with the user's intentions? We outline a high-level research direction to solve the agent alignment problem centered around reward modeling: learning a reward function from interaction with the user and optimizing the learned reward function with reinforcement learning. We discuss the key challenges we expect to face when scaling reward modeling to complex and general domains, concrete approaches to mitigate these challenges, and ways to establish trust in the resulting agents.

Key facts. Cited by 438 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 92 of its 189 references that are in the map. First posted on arXiv 19 Nov 2018.

Builds on

92 of 189 references.

Closest ancestors · 4

Same direction, newest first

  1. 2018Reward learning from human preferences and demonstrations in Atari15 Nov
  2. 2018Learning to Understand Goal Specifications by Modelling Reward5 Jun
  3. 2018Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learning27 May
  4. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 88

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  3. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  4. 2013Intriguing properties of neural networkscited by 5,114
  5. 2017On Calibration of Modern Neural Networkscited by 4,860
  6. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  7. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  8. 2015Continuous control with deep reinforcement learningcited by 3,279
  9. 2016Improved Techniques for Training GANscited by 2,938
  10. 2015Trust Region Policy Optimizationcited by 2,503
  11. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
  12. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,335
38 more
  1. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  2. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  3. 2016Concrete Problems in AI Safetycited by 1,582
  4. 2016Generative Adversarial Imitation Learningcited by 1,576
  5. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  6. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,400
  7. 2018Relational inductive biases, deep learning, and graph networkscited by 1,341
  8. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,309
  9. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,259
  10. 2017Constrained Policy Optimizationcited by 928
  11. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  12. 2017Deep Bayesian Active Learning with Image Datacited by 812
  13. 2017Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networkscited by 739
  14. 2015Weight Uncertainty in Neural Networkscited by 554
  15. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 527
  16. 2015Empirical Evaluation of Rectified Activations in Convolutional Networkcited by 515
  17. 2015Variational Dropout and the Local Reparameterization Trickcited by 489
  18. 2018Accurate Uncertainties for Deep Learning Using Calibrated Regressioncited by 417
  19. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
  20. 2018AI safety via debatecited by 384
  21. 2017FeUdal Networks for Hierarchical Reinforcement Learningcited by 328
  22. 2017Adversarial Attacks on Neural Network Policiescited by 314
  23. 2016Cooperative Inverse Reinforcement Learningcited by 298
  24. 2015Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networkscited by 295
  25. 2018Distributed Prioritized Experience Replaycited by 260
  26. 2018Human-level performance in first-person multiplayer games with population-based deep reinforcement learningcited by 260
  27. 2017Inverse Reward Designcited by 246
  28. 2017Thinking Fast and Slow with Deep Learning and Tree Searchcited by 245
  29. 2017Concrete Dropoutcited by 227
  30. 2018A Note on the Inception Scorecited by 225
  31. 2018Adversarial Risk and the Dangers of Evaluating Against Weak Attackscited by 221
  32. 2018Deep Learning: A Critical Appraisalcited by 214
  33. 2018Distributed Distributional Deterministic Policy Gradientscited by 195
  34. 2018Verifiable Reinforcement Learning via Policy Extractioncited by 176
  35. 2018A Study on Overfitting in Deep Reinforcement Learningcited by 170
  36. 2017Interactive Learning from Policy-Dependent Human Feedbackcited by 155
  37. 2017Visualizing and Understanding Atari Agentscited by 141
  38. 2018Scaling provable adversarial defensescited by 141

50 most cited of 88 All 88 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2019: 2; 2020: 2; 2021: 18; 2022: 34; 2023: 71; 2024: 105; 2025: 125; 2026: 81.

438 papers.

Sort:

By area:

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  3. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  4. 2020Learning to summarize from human feedbackcited by 2,562
  5. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  6. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  7. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  8. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  9. 2023Towards Understanding Sycophancy in Language Modelscited by 988
  10. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  11. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  12. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
38 more
  1. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  2. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  3. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 651
  4. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  5. 2025Process Reinforcement through Implicit Rewardscited by 478
  6. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 450
  7. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  8. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 404
  9. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 394
  10. 2022Self-critiquing models for assisting human evaluatorscited by 372
  11. 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 314
  12. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 298
  13. 2021Unsolved Problems in ML Safetycited by 291
  14. 2022Teaching language models to support answers with verified quotescited by 290
  15. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  16. 2023A Survey of Reinforcement Learning from Human Feedbackcited by 259
  17. 2022The Alignment Problem from a Deep Learning Perspectivecited by 246
  18. 2023Large Language Model Alignment: A Surveycited by 241
  19. 2022Measuring Progress on Scalable Oversight for Large Language Modelscited by 212
  20. 2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judgecited by 202
  21. 2022Defining and Characterizing Reward Hackingcited by 184
  22. 2024A Roadmap to Pluralistic Alignmentcited by 175
  23. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraintscited by 174
  24. 2024Secrets of RLHF in Large Language Models Part II: Reward Modelingcited by 170
  25. 2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 169
  26. 2021Alignment of Language Agentscited by 167
  27. 2024COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllabilitycited by 162
  28. 2024RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedbackcited by 147
  29. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learningcited by 147
  30. 2024LLM Critics Help Catch LLM Bugscited by 138
  31. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  32. 2021Goal Misgeneralization in Deep Reinforcement Learningcited by 122
  33. 2022Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goalscited by 120
  34. 2025WorldModelBench: Judging Video Generation Models As World Modelscited by 116
  35. 2024MLLM-Protector: Ensuring MLLM's Safety without Hurting Performancecited by 111
  36. 2024Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaborationcited by 111
  37. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  38. 2024Controllable Preference Optimization: Toward Controllable Multi-Objective Alignmentcited by 105

50 most cited of 438 All 438 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Reward Models in Deep Reinforcement Learning: A Survey
  2. 2023Behavior Alignment via Reward Function Optimization
  3. 2026Hierarchical Reward Design from Language: Enhancing Alignment of Agent Behavior with Human Specifications
  4. 2026Aligning Agents via Planning: A Benchmark for Trajectory-Level Reward Modeling
  5. 2025Similarity as Reward Alignment: Robust and Versatile Preference-based Reinforcement Learning
  6. 2025Reinforcement Learning for Large Model: A Survey
  7. 2025A Survey on Progress in LLM Alignment from the Perspective of Reward Design
  8. 2025Multi-Agent Collaborative Reward Design for Enhancing Reasoning in Reinforcement Learning