mlsanityData to 14 Oct 2026

DQN-TAMER: Human-in-the-Loop Reinforcement Learning with Intractable Feedback

arXiv 1810.11748 ↗First posted 28 Oct 2018cs.HC cs.LGcited by 41

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Exploration has been one of the greatest challenges in reinforcement learning (RL), which is a large obstacle in the application of RL to robotics. Even with state-of-the-art RL algorithms, building a well-learned agent often requires too many trials, mainly due to the difficulty of matching its actions with rewards in the distant future. A remedy for this is to train an agent with real-time feedback from a human observer who immediately gives rewards for some actions. This study tackles a series of challenges for introducing such a human-in-the-loop RL scheme. The first contribution of this work is our experiments with a precisely modeled human observer: binary, delay, stochasticity, unsustainability, and natural reaction. We also propose an RL method called DQN-TAMER, which efficiently uses both human feedback and distant rewards. We find that DQN-TAMER agents outperform their baselines in Maze and Taxi simulated environments. Furthermore, we demonstrate a real-world human-in-the-loop RL application where a camera automatically recognizes a user's facial expressions as feedback to the agent while the agent explores a maze.

Key facts. Cited by 41 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 10 of its 34 references that are in the map. First posted on arXiv 28 Oct 2018.

Builds on

10 of 34 references.

Closest ancestors · 1

Same direction, newest first

  1. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 9

  1. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  2. 2015End-to-End Training of Deep Visuomotor Policiescited by 875
  3. 2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 353
  4. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 340
  5. 2018RUDDER: Return Decomposition for Delayed Rewardscited by 173
  6. 2017Interactive Learning from Policy-Dependent Human Feedbackcited by 158
  7. 2017Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spacescited by 123
  8. 2016Generalizing Skills with Semi-Supervised Reinforcement Learningcited by 18
  9. 2017MicroExpNet: An Extremely Small and Fast Model For Expression Recognition From Face Imagescited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2020: 2; 2021: 4; 2022: 3; 2023: 10; 2024: 13; 2025: 7; 2026: 2.

41 papers.

Sort:

By area:

  1. 2023Human Preference Score: Better Aligning Text-to-Image Models with Human Preferencecited by 287
  2. 2023Is RLHF More Difficult than Standard RL?cited by 96
  3. 2020The EMPATHIC Framework for Task Learning from Implicit Human Feedbackcited by 38
  4. 2023Accelerating Reinforcement Learning of Robotic Manipulations via Feedback from Large Language Modelscited by 22
  5. 2024Text2Chart31: Instruction Tuning for Chart Generation with Automatic Feedbackcited by 17
  6. 2023Provably Feedback-Efficient Reinforcement Learning via Active Reward Learningcited by 15
  7. 2023Primitive Skill-based Robot Learning from Human Evaluative Feedbackcited by 15
  8. 2021Recent Advances in Leveraging Human Guidance for Sequential Decision-Making Taskscited by 13
  9. 2023Active Reward Learning from Online Preferencescited by 13
  10. 2024CREW: Facilitating Human-AI Teaming Researchcited by 11
  11. 2023Decision Making for Human-in-the-loop Robotic Agents via Uncertainty-Aware Reinforcement Learningcited by 9
  12. 2023Generative AI Meets Future Cities: Towards an Era of Autonomous Urban Intelligencecited by 8
29 more
  1. 2023From "Thumbs Up" to "10 out of 10": Reconsidering Scalar Feedback in Interactive Reinforcement Learningcited by 6
  2. 2022Advice Conformance Verification by Reinforcement Learning agents for Human-in-the-Loopcited by 5
  3. 2024Mapping out the Space of Human Feedback for Reinforcement Learning: A Conceptual Frameworkcited by 5
  4. 2020Accelerating Reinforcement Learning Agent with EEG-based Implicit Human Feedbackcited by 4
  5. 2021Influencing Reinforcement Learning through Natural Language Guidancecited by 4
  6. 2022Towards customizable reinforcement learning agents: Enabling preference specification through online vocabulary expansioncited by 4
  7. 2024Vocal Sandbox: Continual Learning and Adaptation for Situated Human-Robot Collaborationcited by 4
  8. 2025Interactive Groupwise Comparison for Reinforcement Learning from Human Feedbackcited by 4
  9. 2025A Review On Safe Reinforcement Learning Using Lyapunov and Barrier Functionscited by 4
  10. 2021Accelerating the Learning of TAMER with Counterfactual Explanationscited by 3
  11. 2024Opinion-Guided Reinforcement Learningcited by 3
  12. 2024GUIDE: Real-Time Human-Shaped Agentscited by 3
  13. 2024Towards Sample-Efficiency and Generalization of Transfer and Inverse Reinforcement Learning: A Comprehensive Literature Reviewcited by 3
  14. 2021Towards Interactive Reinforcement Learning with Intrinsic Feedbackcited by 2
  15. 2022Incorporating Voice Instructions in Model-Based Reinforcement Learning for Self-Driving Carscited by 2
  16. 2023Human-AI Collaboration in Real-World Complex Environment with Reinforcement Learningcited by 2
  17. 2024How Much Progress Did I Make? An Unexplored Human Feedback Signal for Teaching Robotscited by 2
  18. 2024Evolutionary Retrofittingcited by 2
  19. 2025Contextual bandits with entropy-based human feedbackcited by 1
  20. 2025Towards Reinforcement Learning from Neural Feedback: Mapping fNIRS Signals to Agent Performancecited by 1
  21. 2024Integrating Human Expertise in Continuous Spaces: A Novel Interactive Bayesian Optimization Framework with Preference Expected Improvementcited by 0
  22. 2024The Power of Combined Modalities in Interactive Robot Learningcited by 0
  23. 2024A Digital Twin Framework for Reinforcement Learning with Real-Time Self-Improvement via Human Assistive Teleoperationcited by 0
  24. 2024CrowdTransfer: Enabling Crowd Knowledge Transfer in AIoT Communitycited by 0
  25. 2025Interactive Double Deep Q-network: Integrating Human Interventions and Evaluative Predictions in Reinforcement Learning of Autonomous Drivingcited by 0
  26. 2025CueLearner: Bootstrapping and local policy adaptation from relative feedbackcited by 0
  27. 2025Active Query Selection for Crowd-Based Reinforcement Learningcited by 0
  28. 2026Themis: An explainable AI-enabled framework for Reinforcement Learning with Human Feedbackcited by 0
  29. 2026An offline approach to fNIRS-guided reinforcement learning for robot behaviorcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Zero-Shot LLMs in Human-in-the-Loop RL: Replacing Human Feedback for Reward Shaping
  2. 2023Breadcrumbs to the Goal: Goal-Conditioned Exploration from Human-in-the-Loop Feedback
  3. 2023A Survey of Reinforcement Learning from Human Feedback
  4. 2025Reinforcement Learning from Human Feedback
  5. 2024A Survey On Enhancing Reinforcement Learning in Complex Environments: Insights from Human and LLM Feedback
  6. 2024Leveraging Sub-Optimal Data for Human-in-the-Loop Reinforcement Learning
  7. 2024RLHF Workflow: From Reward Modeling to Online RLHF
  8. 2024SERL: A Software Suite for Sample-Efficient Robotic Reinforcement Learning