mlsanityData to 14 Oct 2026

Deep Reinforcement Learning from Policy-Dependent Human Feedback

arXiv 1902.04257 ↗First posted 12 Feb 2019cs.LG stat.MLcited by 56

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

To widen their accessibility and increase their utility, intelligent agents must be able to learn complex behaviors as specified by (non-expert) human users. Moreover, they will need to learn these behaviors within a reasonable amount of time while efficiently leveraging the sparse feedback a human trainer is capable of providing. Recent work has shown that human feedback can be characterized as a critique of an agent's current behavior rather than as an alternative reward signal to be maximized, culminating in the COnvergent Actor-Critic by Humans (COACH) algorithm for making direct policy updates based on human feedback. Our work builds on COACH, moving to a setting where the agent's policy is represented by a deep neural network. We employ a series of modifications on top of the original COACH algorithm that are critical for successfully learning behaviors from high-dimensional observations, while also satisfying the constraint of obtaining reduced sample complexity. We demonstrate the effectiveness of our Deep COACH algorithm in the rich 3D world of Minecraft with an agent that learns to complete tasks by mapping from raw pixels to actions using only real-time human feedback in 10-15 minutes of interaction.

Key facts. Cited by 56 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 12 of its 41 references that are in the map. First posted on arXiv 12 Feb 2019.

Builds on

12 of 41 references.

Closest ancestors · 1

Same direction, newest first

  1. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 11

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2015Continuous control with deep reinforcement learningcited by 3,407
  4. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  5. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  6. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 406
  7. 2017Interactive Learning from Policy-Dependent Human Feedbackcited by 158
  8. 2017Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spacescited by 123
  9. 2016A Deep Hierarchical Approach to Lifelong Learning in Minecraftcited by 90
  10. 2017Environment-Independent Task Specifications via GLTLcited by 67
  11. 2016Control of Memory, Active Perception, and Action in Minecraftcited by 61

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 10; 2022: 9; 2023: 14; 2024: 12; 2025: 9; 2026: 2.

56 papers.

Sort:

By area:

  1. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  2. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 345
  3. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 302
  4. 2023A Survey of Reinforcement Learning from Human Feedbackcited by 264
  5. 2023Vision-Language Models as Success Detectorscited by 142
  6. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  7. 2023Motif: Intrinsic Motivation from Artificial Intelligence Feedbackcited by 102
  8. 2025Reinforcement Learning Optimization for Large-Scale Learning: An Efficient and User-Friendly Scaling Librarycited by 101
  9. 2021Imitation Learning: Progress, Taxonomies and Challengescited by 58
  10. 2024A Comprehensive Survey of Foundation Models in Medicinecited by 56
  11. 2022Efficient Learning of Safe Driving Policy via Human-AI Copilot Optimizationcited by 49
  12. 2021Skill Preferences: Learning to Extract and Execute Robotic Skills from Human Feedbackcited by 35
38 more
  1. 2023Beyond Reward: Offline Preference-guided Policy Optimizationcited by 35
  2. 2023MCU: An Evaluation Framework for Open-Ended Game Agentscited by 34
  3. 2021Cognitive science as a source of forward and inverse models of human decisions for robotics and controlcited by 22
  4. 2022Continual Learning for Instruction Following from Realtime Feedbackcited by 22
  5. 2023Accelerating Reinforcement Learning of Robotic Manipulations via Feedback from Large Language Modelscited by 22
  6. 2024Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedbackcited by 21
  7. 2024MAP: Multi-Human-Value Alignment Palettecited by 20
  8. 2021Continual Learning for Grounded Instruction Generation by Observing Human Following Behaviorcited by 19
  9. 2024Text2Chart31: Instruction Tuning for Chart Generation with Automatic Feedbackcited by 17
  10. 2023Primitive Skill-based Robot Learning from Human Evaluative Feedbackcited by 15
  11. 2025Enhancing Rating-Based Reinforcement Learning to Effectively Leverage Feedback from Large Vision-Language Modelscited by 15
  12. 2021Recent Advances in Leveraging Human Guidance for Sequential Decision-Making Taskscited by 13
  13. 2023Active Reward Learning from Multiple Teacherscited by 13
  14. 2024CREW: Facilitating Human-AI Teaming Researchcited by 11
  15. 2024Sample-Efficient Preference-based Reinforcement Learning with Dynamics Aware Rewardscited by 9
  16. 2022ASHA: Assistive Teleoperation via Human-in-the-Loop Reinforcement Learningcited by 8
  17. 2023Shattering the Agent-Environment Interface for Fine-Tuning Inclusive Language Modelscited by 8
  18. 2023BEDD: The MineRL BASALT Evaluation and Demonstrations Dataset for Training and Benchmarking Agents that Solve Fuzzy Taskscited by 8
  19. 2025Reveal the Mystery of DPO: The Connection between DPO and RL Algorithmscited by 8
  20. 2025An Outlook on the Opportunities and Challenges of Multi-Agent AI Systemscited by 8
  21. 2022X2T: Training an X-to-Text Typing Interface with Online Learning from User Feedbackcited by 4
  22. 2022Rewards Encoding Environment Dynamics Improves Preference-based Reinforcement Learningcited by 4
  23. 2022Discovering Generalizable Spatial Goal Representations via Graph-based Active Reward Learningcited by 3
  24. 2024CANDERE-COACH: Reinforcement Learning from Noisy Feedbackcited by 3
  25. 2024GUIDE: Real-Time Human-Shaped Agentscited by 3
  26. 2024Guiding Reinforcement Learning Using Uncertainty-Aware Large Language Modelscited by 3
  27. 2021GAN-Based Interactive Reinforcement Learning from Demonstration and Human Evaluative Feedbackcited by 2
  28. 2021Towards Interactive Reinforcement Learning with Intrinsic Feedbackcited by 2
  29. 2022Teachable Reinforcement Learning via Advice Distillationcited by 2
  30. 2022Incorporating Voice Instructions in Model-Based Reinforcement Learning for Self-Driving Carscited by 2
  31. 2023Human-AI Collaboration in Real-World Complex Environment with Reinforcement Learningcited by 2
  32. 2024How Much Progress Did I Make? An Unexplored Human Feedback Signal for Teaching Robotscited by 2
  33. 2025LAPP: Large Language Model Feedback for Preference-Driven Reinforcement Learningcited by 2
  34. 2021Learning Online from Corrective Feedback: A Meta-Algorithm for Roboticscited by 1
  35. 2023STRAPPER: Preference-based Reinforcement Learning via Self-training Augmentation and Peer Regularizationcited by 1
  36. 2025PEO: Improving Bi-Factorial Preference Alignment with Post-Training Policy Extrapolationcited by 1
  37. 2025Pref-GUIDE: Continual Policy Learning from Real-Time Human Feedback via Preference-Based Learningcited by 1
  38. 2026AirSplat: Alignment and Rating for Robust Feed-Forward 3D Gaussian Splattingcited by 1

50 most cited of 56 All 56 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2017Explore, Exploit or Listen: Combining Human Feedback and Policy Model to Speed up Deep Reinforcement Learning in 3D Worlds
  2. 2021Combining Learning from Human Feedback and Knowledge Engineering to Solve Hierarchical Tasks in Minecraft
  3. 2025An Introduction to Deep Reinforcement and Imitation Learning
  4. 2024Mapping out the Space of Human Feedback for Reinforcement Learning: A Conceptual Framework
  5. 2019Leveraging Human Guidance for Deep Reinforcement Learning Tasks
  6. 2024Zero-shot Imitation Policy via Search in Demonstration Dataset
  7. 2025Policy-Conditioned Policies for Multi-Agent Task Solving
  8. 2022Improving Multimodal Interactive Agents with Reinforcement Learning from Human Feedback