mlsanityData to 14 Oct 2026

Deep reinforcement learning from human preferences

arXiv 1706.03741 ↗First posted 12 Jun 2017stat.ML cs.AI cs.HCcited by 3,912

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

For sophisticated reinforcement learning (RL) systems to interact usefully with real-world environments, we need to communicate complex goals to these systems. In this work, we explore goals defined in terms of (non-expert) human preferences between pairs of trajectory segments. We show that this approach can effectively solve complex RL tasks without access to the reward function, including Atari games and simulated robot locomotion, while providing feedback on less than one percent of our agent's interactions with the environment. This reduces the cost of human oversight far enough that it can be practically applied to state-of-the-art RL systems. To demonstrate the flexibility of our approach, we show that we can successfully train complex novel behaviors with about an hour of human time. These behaviors and environments are considerably more complex than any that have been previously learned from human feedback.

Key facts. Cited by 3,912 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 13 of its 45 references that are in the map. First posted on arXiv 12 Jun 2017.

Builds on

13 of 45 references.

References · 13

  1. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  2. 2015Trust Region Policy Optimizationcited by 2,580
  3. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  4. 2016Concrete Problems in AI Safetycited by 1,642
  5. 2016Generative Adversarial Imitation Learningcited by 1,597
  6. 2016TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systemscited by 664
  7. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 406
  8. 2016Cooperative Inverse Reinforcement Learningcited by 300
  9. 2017Interactive Learning from Policy-Dependent Human Feedbackcited by 158
  10. 2017Third-Person Imitation Learningcited by 103
  11. 2016Learning Language Games through Interactioncited by 43
  12. 2017Deep Q-learning from Demonstrationscited by 28
1 more
  1. 2016Generalizing Skills with Semi-Supervised Reinforcement Learningcited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2017: 1; 2018: 6; 2019: 12; 2020: 15; 2021: 73; 2022: 111; 2023: 594; 2024: 927; 2025: 1095; 2026: 1078.

3,912 papers.

Sort:

By area:

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2023Let's Verify Step by Stepcited by 3,789
  6. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  7. 2023Qwen Technical Reportcited by 3,344
  8. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  9. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  10. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  11. 2020Learning to summarize from human feedbackcited by 2,588
  12. 2023A Survey of Large Language Modelscited by 2,576
38 more
  1. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  2. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
  3. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  4. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  5. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  6. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,395
  7. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,345
  8. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  9. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  10. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  11. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  12. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,045
  13. 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
  14. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 991
  15. 2023Training Diffusion Models with Reinforcement Learningcited by 982
  16. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 977
  17. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  18. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 916
  19. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  20. 2022Solving math word problems with process- and outcome-based feedbackcited by 901
  21. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 894
  22. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
  23. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
  24. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 783
  25. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 767
  26. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  27. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  28. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 729
  29. 2023DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvaturecited by 726
  30. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 725
  31. 2023A Watermark for Large Language Modelscited by 725
  32. 2020Aligning AI With Shared Human Valuescited by 723
  33. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
  34. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 703
  35. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 672
  36. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  37. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 638
  38. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 625

50 most cited of 3,912 All 3,912 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2020Weak Human Preference Supervision For Deep Reinforcement Learning
  2. 2023A Survey of Reinforcement Learning from Human Feedback
  3. 2024Mapping out the Space of Human Feedback for Reinforcement Learning: A Conceptual Framework
  4. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  5. 2024A Survey On Enhancing Reinforcement Learning in Complex Environments: Insights from Human and LLM Feedback
  6. 2024PrefCLM: Enhancing Preference-based Reinforcement Learning with Crowdsourced Large Language Models ↗
  7. 2025Reinforcement Learning from Human Feedback
  8. 2019Leveraging Human Guidance for Deep Reinforcement Learning Tasks