# Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spaces

arXiv: https://arxiv.org/abs/1709.10163 · HTML: https://mlsanity.com/paper/1709.10163/

## Key facts

- Cited by 123 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Sep 2017.

## Built on by (most cited first, 30 of 123)

- [ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation](https://mlsanity.com/paper/2304.05977/) (2023)
- [Scalable agent alignment via reward modeling: a research direction](https://mlsanity.com/paper/1811.07871/) (2018)
- [Aligning Text-to-Image Models using Human Feedback](https://mlsanity.com/paper/2302.12192/) (2023)
- [Augmented Language Models: a Survey](https://mlsanity.com/paper/2302.07842/) (2023)
- [Reward learning from human preferences and demonstrations in Atari](https://mlsanity.com/paper/1811.06521/) (2018)
- [PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training](https://mlsanity.com/paper/2106.05091/) (2021)
- [Preference Ranking Optimization for Human Alignment](https://mlsanity.com/paper/2306.17492/) (2023)
- [Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons](https://mlsanity.com/paper/2301.11270/) (2023)
- [Chain of Hindsight Aligns Language Models with Feedback](https://mlsanity.com/paper/2302.02676/) (2023)
- [Robot Learning on the Job: Human-in-the-Loop Autonomy and Learning During Deployment](https://mlsanity.com/paper/2211.08416/) (2022)
- [B-Pref: Benchmarking Preference-Based Reinforcement Learning](https://mlsanity.com/paper/2111.03026/) (2021)
- [Reinforcement Learning Optimization for Large-Scale Learning: An Efficient and User-Friendly Scaling Library](https://mlsanity.com/paper/2506.06122/) (2025)
- [Is RLHF More Difficult than Standard RL?](https://mlsanity.com/paper/2306.14111/) (2023)
- [Human Alignment of Large Language Models through Online Preference Optimisation](https://mlsanity.com/paper/2403.08635/) (2024)
- [Few-Shot Preference Learning for Human-in-the-Loop RL](https://mlsanity.com/paper/2212.03363/) (2022)
- [Learning to Understand Goal Specifications by Modelling Reward](https://mlsanity.com/paper/1806.01946/) (2018)
- [RLHF from Heterogeneous Feedback via Personalization and Preference Aggregation](https://mlsanity.com/paper/2405.00254/) (2024)
- [Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHF](https://mlsanity.com/paper/2401.16335/) (2024)
- [Deep Reinforcement Learning from Policy-Dependent Human Feedback](https://mlsanity.com/paper/1902.04257/) (2019)
- [Opening up ChatGPT: Tracking openness, transparency, and accountability in instruction-tuned text generators](https://mlsanity.com/paper/2307.05532/) (2023)
- [Provable Offline Preference-Based Reinforcement Learning](https://mlsanity.com/paper/2305.14816/) (2023)
- [Efficient Learning of Safe Driving Policy via Human-AI Copilot Optimization](https://mlsanity.com/paper/2202.10341/) (2022)
- [Offline Regularised Reinforcement Learning for Large Language Models Alignment](https://mlsanity.com/paper/2405.19107/) (2024)
- [DQN-TAMER: Human-in-the-Loop Reinforcement Learning with Intractable Feedback](https://mlsanity.com/paper/1810.11748/) (2018)
- [Dataset Reset Policy Optimization for RLHF](https://mlsanity.com/paper/2404.08495/) (2024)
- [The EMPATHIC Framework for Task Learning from Implicit Human Feedback](https://mlsanity.com/paper/2009.13649/) (2020)
- [Skill Preferences: Learning to Extract and Execute Robotic Skills from Human Feedback](https://mlsanity.com/paper/2108.05382/) (2021)
- [Beyond Reward: Offline Preference-guided Policy Optimization](https://mlsanity.com/paper/2305.16217/) (2023)
- [AI Alignment with Changing and Influenceable Reward Functions](https://mlsanity.com/paper/2405.17713/) (2024)
- [Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences](https://mlsanity.com/paper/2403.01857/) (2024)

## Abstract (arXiv)

While recent advances in deep reinforcement learning have allowed autonomous learning agents to succeed at a variety of complex tasks, existing algorithms generally require a lot of training data. One way to increase the speed at which agents are able to learn to perform tasks is by leveraging the input of human trainers. Although such input can take many forms, real-time, scalar-valued feedback is especially useful in situations where it proves difficult or impossible for humans to provide expert demonstrations. Previous approaches have shown the usefulness of human input provided in this fashion (e.g., the TAMER framework), but they have thus far not considered high-dimensional state spaces or employed the use of deep learning. In this paper, we do both: we propose Deep TAMER, an extension of the TAMER framework that leverages the representational power of deep neural networks in order to learn complex tasks in just a short amount of time with a human trainer. We demonstrate Deep TAMER's success by using it and just 15 minutes of human-provided feedback to train an agent that performs better than humans on the Atari game of Bowling - a task that has proven difficult for even state-of-the-art reinforcement learning methods.