Data to 7 Oct 2026

Interactive Learning from Policy-Dependent Human Feedback

arXiv 1701.06049 ↗First posted 21 Jan 2017cs.AIcited by 155

Not on the map. Why?

Cited by 68 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

This paper investigates the problem of interactively learning behaviors communicated by a human teacher using positive and negative feedback. Much previous work on this problem has made the assumption that people provide feedback for decisions that is dependent on the behavior they are teaching and is independent from the learner's current policy. We present empirical results that show this assumption to be false -- whether human trainers give a positive or negative feedback for a decision is influenced by the learner's current policy. Based on this insight, we introduce em Convergent Actor-Critic by Humans (COACH), an algorithm for learning from policy-dependent feedback that converges to a local optimum. Finally, we demonstrate that COACH can successfully learn multiple behaviors on a physical robot.

Key facts. Cited by 155 papers in the map (data through 7 Oct 2026). First posted on arXiv 21 Jan 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2017: 2; 2018: 4; 2019: 2; 2020: 4; 2021: 19; 2022: 17; 2023: 39; 2024: 34; 2025: 24; 2026: 10.

155 papers.

Sort:

By area:

2017Deep reinforcement learning from human preferencescited by 3,846
2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
2018Scalable agent alignment via reward modeling: a research directioncited by 438
2023Aligning Text-to-Image Models using Human Feedbackcited by 418
2023Augmented Language Models: a Surveycited by 398
2018Reward learning from human preferences and demonstrations in Ataricited by 314
2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 298
2023Preference Ranking Optimization for Human Alignmentcited by 283
2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 244
2023Secrets of RLHF in Large Language Models Part I: PPOcited by 226

38 more

2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimizationcited by 223
2023Habitat 3.0: A Co-Habitat for Humans, Avatars and Robotscited by 220
2023Chain of Hindsight Aligns Language Models with Feedbackcited by 147
2023Vision-Language Models as Success Detectorscited by 138
2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
2020Human-in-the-Loop Imitation Learning using Remote Teleoperationcited by 103
2022Robot Learning on the Job: Human-in-the-Loop Autonomy and Learning During Deploymentcited by 101
2023Motif: Intrinsic Motivation from Artificial Intelligence Feedbackcited by 99
2023Loose lips sink ships: Mitigating Length Bias in Reinforcement Learning from Human Feedbackcited by 78
2021On the Expressivity of Markov Rewardcited by 71
2018Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learningcited by 57
2024Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHFcited by 57
2019Deep Reinforcement Learning from Policy-Dependent Human Feedbackcited by 56
2024InstructGraph: Boosting Large Language Models via Graph-centric Instruction Tuning and Preference Alignmentcited by 56
2024A Comprehensive Survey of Foundation Models in Medicinecited by 56
2023Provable Offline Preference-Based Reinforcement Learningcited by 55
2024CAT: Enhancing Multimodal Large Language Model to Answer Questions in Dynamic Audio-Visual Scenarioscited by 53
2024Transfer Q Star: Principled Decoding for LLM Alignmentcited by 46
2020Learning Rewards from Linguistic Feedbackcited by 41
2024Dataset Reset Policy Optimization for RLHFcited by 41
2018DQN-TAMER: Human-in-the-Loop Reinforcement Learning with Intractable Feedbackcited by 39
2020The EMPATHIC Framework for Task Learning from Implicit Human Feedbackcited by 38
2021Skill Preferences: Learning to Extract and Execute Robotic Skills from Human Feedbackcited by 34
2024Recent Advances in Generative AI and Large Language Models: Current Status, Challenges, and Perspectivescited by 30
2024Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferencescited by 27
2019Risk-Aware Active Inverse Reinforcement Learningcited by 26
2022Improving Multimodal Interactive Agents with Reinforcement Learning from Human Feedbackcited by 26
2023When to Show a Suggestion? Integrating Human Feedback in AI-Assisted Programmingcited by 25
2021Information Directed Reward Learning for Reinforcement Learningcited by 24
2021Cognitive science as a source of forward and inverse models of human decisions for robotics and controlcited by 22
2022How to talk so AI will learn: Instructions, descriptions, and autonomycited by 22
2023Accelerating Reinforcement Learning of Robotic Manipulations via Feedback from Large Language Modelscited by 22
2021Prioritized Experience-based Reinforcement Learning with Human Guidance for Autonomous Drivingcited by 21
2022Continual Learning for Instruction Following from Realtime Feedbackcited by 21
2022When to Ask for Help: Proactive Interventions in Autonomous Reinforcement Learningcited by 20
2024Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedbackcited by 20
2021Continual Learning for Grounded Instruction Generation by Observing Human Following Behaviorcited by 19
2024Learning to Watermark LLM-generated Text via Reinforcement Learningcited by 19

50 most cited of 155 All 155 (JSON)