mlsanityData to 7 Oct 2026

Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spaces

arXiv 1709.10163 ↗First posted 28 Sep 2017cs.AI cs.LGcited by 122

Not on the map. Why?

Cited by 56 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

While recent advances in deep reinforcement learning have allowed autonomous learning agents to succeed at a variety of complex tasks, existing algorithms generally require a lot of training data. One way to increase the speed at which agents are able to learn to perform tasks is by leveraging the input of human trainers. Although such input can take many forms, real-time, scalar-valued feedback is especially useful in situations where it proves difficult or impossible for humans to provide expert demonstrations. Previous approaches have shown the usefulness of human input provided in this fashion (e.g., the TAMER framework), but they have thus far not considered high-dimensional state spaces or employed the use of deep learning. In this paper, we do both: we propose Deep TAMER, an extension of the TAMER framework that leverages the representational power of deep neural networks in order to learn complex tasks in just a short amount of time with a human trainer. We demonstrate Deep TAMER's success by using it and just 15 minutes of human-provided feedback to train an agent that performs better than humans on the Atari game of Bowling - a task that has proven difficult for even state-of-the-art reinforcement learning methods.

Key facts. Cited by 122 papers in the map (data through 7 Oct 2026). First posted on arXiv 28 Sep 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 4; 2019: 2; 2020: 5; 2021: 12; 2022: 20; 2023: 29; 2024: 28; 2025: 19; 2026: 3.

122 papers.

Sort:

By area:

  1. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  2. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
  3. 2023Aligning Text-to-Image Models using Human Feedbackcited by 418
  4. 2023Augmented Language Models: a Surveycited by 398
  5. 2018Reward learning from human preferences and demonstrations in Ataricited by 314
  6. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 298
  7. 2023Preference Ranking Optimization for Human Alignmentcited by 283
  8. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 244
  9. 2023Chain of Hindsight Aligns Language Models with Feedbackcited by 147
  10. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  11. 2022Robot Learning on the Job: Human-in-the-Loop Autonomy and Learning During Deploymentcited by 101
  12. 2023Is RLHF More Difficult than Standard RL?cited by 96
38 more
  1. 2025Reinforcement Learning Optimization for Large-Scale Learning: An Efficient and User-Friendly Scaling Librarycited by 94
  2. 2024Human Alignment of Large Language Models through Online Preference Optimisationcited by 90
  3. 2022Few-Shot Preference Learning for Human-in-the-Loop RLcited by 85
  4. 2018Learning to Understand Goal Specifications by Modelling Rewardcited by 80
  5. 2024RLHF from Heterogeneous Feedback via Personalization and Preference Aggregationcited by 71
  6. 2024Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHFcited by 57
  7. 2019Deep Reinforcement Learning from Policy-Dependent Human Feedbackcited by 56
  8. 2023Opening up ChatGPT: Tracking openness, transparency, and accountability in instruction-tuned text generatorscited by 56
  9. 2023Provable Offline Preference-Based Reinforcement Learningcited by 55
  10. 2024Offline Regularised Reinforcement Learning for Large Language Models Alignmentcited by 48
  11. 2022Efficient Learning of Safe Driving Policy via Human-AI Copilot Optimizationcited by 44
  12. 2024Dataset Reset Policy Optimization for RLHFcited by 41
  13. 2018DQN-TAMER: Human-in-the-Loop Reinforcement Learning with Intractable Feedbackcited by 39
  14. 2020The EMPATHIC Framework for Task Learning from Implicit Human Feedbackcited by 38
  15. 2023Beyond Reward: Offline Preference-guided Policy Optimizationcited by 35
  16. 2021Skill Preferences: Learning to Extract and Execute Robotic Skills from Human Feedbackcited by 34
  17. 2024AI Alignment with Changing and Influenceable Reward Functionscited by 33
  18. 2024Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferencescited by 27
  19. 2021The MineRL BASALT Competition on Learning from Human Feedbackcited by 24
  20. 2021MORAL: Aligning AI with Human Norms through Multi-Objective Reinforced Active Learningcited by 24
  21. 2023Accelerating Reinforcement Learning of Robotic Manipulations via Feedback from Large Language Modelscited by 22
  22. 2022Continual Learning for Instruction Following from Realtime Feedbackcited by 21
  23. 2024Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedbackcited by 20
  24. 2021Continual Learning for Grounded Instruction Generation by Observing Human Following Behaviorcited by 19
  25. 2025Detecting Proxy Gaming in RL and LLM Alignment via Evaluator Stress Testscited by 19
  26. 2024Principled Preferential Bayesian Optimizationcited by 16
  27. 2025Learning from Active Human Involvement through Proxy Value Propagationcited by 16
  28. 2023Primitive Skill-based Robot Learning from Human Evaluative Feedbackcited by 15
  29. 2023RLHF-Blender: A Configurable Interactive Interface for Learning from Diverse Human Feedbackcited by 15
  30. 2025Enhancing Rating-Based Reinforcement Learning to Effectively Leverage Feedback from Large Vision-Language Modelscited by 15
  31. 2023Rating-based Reinforcement Learningcited by 14
  32. 2021Recent Advances in Leveraging Human Guidance for Sequential Decision-Making Taskscited by 13
  33. 2023Breadcrumbs to the Goal: Goal-Conditioned Exploration from Human-in-the-Loop Feedbackcited by 13
  34. 2023Active Reward Learning from Online Preferencescited by 12
  35. 2024Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inferencecited by 12
  36. 2023Aligning Robot and Human Representationscited by 11
  37. 2023Continually Improving Extractive QA via Human Feedbackcited by 11
  38. 2024CREW: Facilitating Human-AI Teaming Researchcited by 11

50 most cited of 122 All 122 (JSON)