# DQN-TAMER: Human-in-the-Loop Reinforcement Learning with Intractable Feedback

arXiv: https://arxiv.org/abs/1810.11748 · HTML: https://mlsanity.com/paper/1810.11748/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 10 of its 34 references that are in the map.
- First posted on arXiv 28 Oct 2018.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Reward modeling and RLHF](https://mlsanity.com/alignment/reward-modeling-and-rlhf/)

## Builds on (most cited first, 10 of 10)

- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [End-to-End Training of Deep Visuomotor Policies](https://mlsanity.com/paper/1504.00702/) (2015)
- [Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updates](https://mlsanity.com/paper/1610.00633/) (2016)
- [Overcoming Exploration in Reinforcement Learning with Demonstrations](https://mlsanity.com/paper/1709.10089/) (2017)
- [RUDDER: Return Decomposition for Delayed Rewards](https://mlsanity.com/paper/1806.07857/) (2018)
- [Interactive Learning from Policy-Dependent Human Feedback](https://mlsanity.com/paper/1701.06049/) (2017)
- [Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spaces](https://mlsanity.com/paper/1709.10163/) (2017)
- [Generalizing Skills with Semi-Supervised Reinforcement Learning](https://mlsanity.com/paper/1612.00429/) (2016)
- [MicroExpNet: An Extremely Small and Fast Model For Expression Recognition From Face Images](https://mlsanity.com/paper/1711.07011/) (2017)

## Built on by (most cited first, 30 of 41)

- [Human Preference Score: Better Aligning Text-to-Image Models with Human Preference](https://mlsanity.com/paper/2303.14420/) (2023)
- [Is RLHF More Difficult than Standard RL?](https://mlsanity.com/paper/2306.14111/) (2023)
- [The EMPATHIC Framework for Task Learning from Implicit Human Feedback](https://mlsanity.com/paper/2009.13649/) (2020)
- [Accelerating Reinforcement Learning of Robotic Manipulations via Feedback from Large Language Models](https://mlsanity.com/paper/2311.02379/) (2023)
- [Text2Chart31: Instruction Tuning for Chart Generation with Automatic Feedback](https://mlsanity.com/paper/2410.04064/) (2024)
- [Provably Feedback-Efficient Reinforcement Learning via Active Reward Learning](https://mlsanity.com/paper/2304.08944/) (2023)
- [Primitive Skill-based Robot Learning from Human Evaluative Feedback](https://mlsanity.com/paper/2307.15801/) (2023)
- [Recent Advances in Leveraging Human Guidance for Sequential Decision-Making Tasks](https://mlsanity.com/paper/2107.05825/) (2021)
- [Active Reward Learning from Online Preferences](https://mlsanity.com/paper/2302.13507/) (2023)
- [CREW: Facilitating Human-AI Teaming Research](https://mlsanity.com/paper/2408.00170/) (2024)
- [Decision Making for Human-in-the-loop Robotic Agents via Uncertainty-Aware Reinforcement Learning](https://mlsanity.com/paper/2303.06710/) (2023)
- [Generative AI Meets Future Cities: Towards an Era of Autonomous Urban Intelligence](https://mlsanity.com/paper/2304.03892/) (2023)
- [From "Thumbs Up" to "10 out of 10": Reconsidering Scalar Feedback in Interactive Reinforcement Learning](https://mlsanity.com/paper/2311.10284/) (2023)
- [Advice Conformance Verification by Reinforcement Learning agents for Human-in-the-Loop](https://mlsanity.com/paper/2210.03455/) (2022)
- [Mapping out the Space of Human Feedback for Reinforcement Learning: A Conceptual Framework](https://mlsanity.com/paper/2411.11761/) (2024)
- [Accelerating Reinforcement Learning Agent with EEG-based Implicit Human Feedback](https://mlsanity.com/paper/2006.16498/) (2020)
- [Influencing Reinforcement Learning through Natural Language Guidance](https://mlsanity.com/paper/2104.01506/) (2021)
- [Towards customizable reinforcement learning agents: Enabling preference specification through online vocabulary expansion](https://mlsanity.com/paper/2210.15096/) (2022)
- [Vocal Sandbox: Continual Learning and Adaptation for Situated Human-Robot Collaboration](https://mlsanity.com/paper/2411.02599/) (2024)
- [Interactive Groupwise Comparison for Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2507.04340/) (2025)
- [A Review On Safe Reinforcement Learning Using Lyapunov and Barrier Functions](https://mlsanity.com/paper/2508.09128/) (2025)
- [Accelerating the Learning of TAMER with Counterfactual Explanations](https://mlsanity.com/paper/2108.01358/) (2021)
- [Opinion-Guided Reinforcement Learning](https://mlsanity.com/paper/2405.17287/) (2024)
- [GUIDE: Real-Time Human-Shaped Agents](https://mlsanity.com/paper/2410.15181/) (2024)
- [Towards Sample-Efficiency and Generalization of Transfer and Inverse Reinforcement Learning: A Comprehensive Literature Review](https://mlsanity.com/paper/2411.10268/) (2024)
- [Towards Interactive Reinforcement Learning with Intrinsic Feedback](https://mlsanity.com/paper/2112.01575/) (2021)
- [Incorporating Voice Instructions in Model-Based Reinforcement Learning for Self-Driving Cars](https://mlsanity.com/paper/2206.10249/) (2022)
- [Human-AI Collaboration in Real-World Complex Environment with Reinforcement Learning](https://mlsanity.com/paper/2312.15160/) (2023)
- [How Much Progress Did I Make? An Unexplored Human Feedback Signal for Teaching Robots](https://mlsanity.com/paper/2407.06459/) (2024)
- [Evolutionary Retrofitting](https://mlsanity.com/paper/2410.11330/) (2024)

## Abstract (arXiv)

Exploration has been one of the greatest challenges in reinforcement learning (RL), which is a large obstacle in the application of RL to robotics. Even with state-of-the-art RL algorithms, building a well-learned agent often requires too many trials, mainly due to the difficulty of matching its actions with rewards in the distant future. A remedy for this is to train an agent with real-time feedback from a human observer who immediately gives rewards for some actions. This study tackles a series of challenges for introducing such a human-in-the-loop RL scheme. The first contribution of this work is our experiments with a precisely modeled human observer: binary, delay, stochasticity, unsustainability, and natural reaction. We also propose an RL method called DQN-TAMER, which efficiently uses both human feedback and distant rewards. We find that DQN-TAMER agents outperform their baselines in Maze and Taxi simulated environments. Furthermore, we demonstrate a real-world human-in-the-loop RL application where a camera automatically recognizes a user's facial expressions as feedback to the agent while the agent explores a maze.