mlsanityData to 7 Oct 2026

Reward learning from human preferences and demonstrations in Atari

arXiv 1811.06521 ↗First posted 15 Nov 2018cs.LG cs.AI cs.NEcited by 314

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

To solve complex real-world problems with reinforcement learning, we cannot rely on manually specified reward functions. Instead, we can have humans communicate an objective to the agent directly. In this work, we combine two approaches to learning from human feedback: expert demonstrations and trajectory preferences. We train a deep neural network to model the reward function and use its predicted reward to train an DQN-based deep reinforcement learning agent on 9 Atari games. Our approach beats the imitation learning baseline in 7 games and achieves strictly superhuman performance on 2 games without using game rewards. Additionally, we investigate the goodness of fit of the reward model, present some reward hacking problems, and study the effects of noise in the human labels.

Key facts. Cited by 314 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 32 of its 60 references that are in the map. First posted on arXiv 15 Nov 2018.

Builds on

32 of 60 references.

Closest ancestors · 3

Same direction, newest first

  1. 2017Explore, Exploit or Listen: Combining Human Feedback and Policy Model to Speed up Deep Reinforcement Learning in 3D Worlds12 Sep
  2. 2017Deep reinforcement learning from human preferences12 Jun
  3. 2017Actor-Critic Reinforcement Learning with Simultaneous Human Control and Feedback3 Mar

Other references · 29

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  5. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  6. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  7. 2016Concrete Problems in AI Safetycited by 1,582
  8. 2016Generative Adversarial Imitation Learningcited by 1,576
  9. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  10. 2017Hindsight Experience Replaycited by 1,096
  11. 2015Prioritized Experience Replaycited by 1,070
  12. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769
17 more
  1. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  2. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  3. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  4. 2017Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning resultscited by 470
  5. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  6. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 400
  7. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 322
  8. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 302
  9. 2016Variational Intrinsic Controlcited by 262
  10. 2017Interactive Learning from Policy-Dependent Human Feedbackcited by 155
  11. 2015Variational Information Maximisation for Intrinsically Motivated Reinforcement Learningcited by 142
  12. 2018Reinforcement and Imitation Learning for Diverse Visuomotor Skillscited by 124
  13. 2017Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spacescited by 122
  14. 2017Trial without Error: Towards Safe Reinforcement Learning via Human Interventioncited by 102
  15. 2018The Surprising Creativity of Digital Evolution: A Collection of Anecdotes from the Evolutionary Computation and Artificial Life Research Communitiescited by 78
  16. 2013Empowerment -- an Introductioncited by 65
  17. 2018Pretraining Deep Actor-Critic Reinforcement Learning Algorithms With Expert Demonstrationscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 1; 2019: 6; 2020: 9; 2021: 33; 2022: 32; 2023: 75; 2024: 71; 2025: 55; 2026: 32.

314 papers.

Sort:

By area:

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2020Learning to summarize from human feedbackcited by 2,562
  3. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  4. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  5. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  6. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  7. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  8. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  9. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
  10. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewardscited by 431
  11. 2023Aligning Text-to-Image Models using Human Feedbackcited by 418
  12. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
38 more
  1. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 298
  2. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  3. 2023Reward Design with Language Modelscited by 257
  4. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 256
  5. 2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 225
  6. 2022Optimizing Prompts for Text-to-Image Generationcited by 204
  7. 2022Fine-tuning language models to find agreement among humans with diverse preferencescited by 189
  8. 2022Defining and Characterizing Reward Hackingcited by 184
  9. 2023HIVE: Harnessing Human Feedback for Instructional Visual Editingcited by 172
  10. 2021Alignment of Language Agentscited by 167
  11. 2023Text2Reward: Reward Shaping with Language Models for Reinforcement Learningcited by 153
  12. 2024RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedbackcited by 147
  13. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedbackcited by 141
  14. 2021Collaborating with Humans without Human Datacited by 139
  15. 2023Vision-Language Models as Success Detectorscited by 138
  16. 2021Goal Misgeneralization in Deep Reinforcement Learningcited by 122
  17. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  18. 2021Dueling RL: Reinforcement Learning with Trajectory Preferencescited by 105
  19. 2021Reward (Mis)design for Autonomous Drivingcited by 102
  20. 2021ThriftyDAgger: Budget-Aware Novelty and Risk Gating for Interactive Imitation Learningcited by 98
  21. 2020Toward the Fundamental Limits of Imitation Learningcited by 96
  22. 2022SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learningcited by 96
  23. 2023Is RLHF More Difficult than Standard RL?cited by 96
  24. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 89
  25. 2022A data-driven approach for learning to control computerscited by 87
  26. 2023Preference Transformer: Modeling Human Preferences using Transformers for RLcited by 77
  27. 2023Contrastive Preference Learning: Learning from Human Feedback without RLcited by 77
  28. 2024InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modelingcited by 76
  29. 2020Safe Imitation Learning via Fast Bayesian Reward Inference from Preferencescited by 73
  30. 2020Active Preference-Based Gaussian Process Regression for Reward Learningcited by 70
  31. 2022Adversarial Motion Priors Make Good Substitutes for Complex Reward Functionscited by 68
  32. 2023Inverse Preference Learning: Preference-based RL without a Reward Functioncited by 68
  33. 2022Reward Uncertainty for Exploration in Preference-based Reinforcement Learningcited by 60
  34. 2023The Wisdom of Hindsight Makes Language Models Better Instruction Followerscited by 59
  35. 2021Imitation Learning: Progress, Taxonomies and Challengescited by 56
  36. 2023Benchmarks and Algorithms for Offline Preference-Based Reward Learningcited by 55
  37. 2024A Dense Reward View on Aligning Text-to-Image Diffusion with Preferencecited by 55
  38. 2022Models of human preference for learning reward functionscited by 54

50 most cited of 314 All 314 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Learning Guarantee of Reward Modeling Using Deep Neural Networks
  2. 2018Inverse reinforcement learning for video games
  3. 2026Training a Conditioned Video Game Agent on a VLM Annotated Dataset
  4. 2020Weak Human Preference Supervision For Deep Reinforcement Learning
  5. 2025Learning To Explore With Predictive World Model Via Self-Supervised Learning
  6. 2020Reward Shaping for Human Learning via Inverse Reinforcement Learning
  7. 2025An Introduction to Deep Reinforcement and Imitation Learning
  8. 2019Deep Reinforcement Learning from Policy-Dependent Human Feedback