mlsanityData to 7 Oct 2026

Safe Deep RL in 3D Environments using Human Feedback

arXiv 2201.08102 ↗First posted 20 Jan 2022cs.LGcited by 5

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Agents should avoid unsafe behaviour during both training and deployment. This typically requires a simulator and a procedural specification of unsafe behaviour. Unfortunately, a simulator is not always available, and procedurally specifying constraints can be difficult or impossible for many real-world tasks. A recently introduced technique, ReQueST, aims to solve this problem by learning a neural simulator of the environment from safe human trajectories, then using the learned simulator to efficiently learn a reward model from human feedback. However, it is yet unknown whether this approach is feasible in complex 3D environments with feedback obtained from real humans - whether sufficient pixel-based neural simulator quality can be achieved, and whether the human data requirements are viable in terms of both quantity and quality. In this paper we answer this question in the affirmative, using ReQueST to train an agent to perform a 3D first-person object collection task using data entirely from human contractors. We show that the resulting agent exhibits an order of magnitude reduction in unsafe behaviour compared to standard reinforcement learning.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 36 of its 53 references that are in the map. First posted on arXiv 20 Jan 2022.

Builds on

36 of 53 references.

Closest ancestors · 6

Same direction, newest first

  1. 2020Semi-supervised reward learning for offline reinforcement learning12 Dec
  2. 2019Learning Human Objectives by Evaluating Hypothetical Behavior5 Dec
  3. 2018Scalable agent alignment via reward modeling: a research direction19 Nov
  4. 2018Reward learning from human preferences and demonstrations in Atari15 Nov
  5. 2018Learning to Understand Goal Specifications by Modelling Reward5 Jun
  6. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 30

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  4. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,967
  5. 2020Learning to summarize from human feedbackcited by 2,562
  6. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  7. 2017Constrained Policy Optimizationcited by 928
  8. 2018Recurrent World Models Facilitate Policy Evolutioncited by 761
  9. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  10. 2020Scaling Laws for Autoregressive Generative Modelingcited by 510
  11. 2017Safe Reinforcement Learning via Shieldingcited by 427
  12. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
18 more
  1. 2018AI safety via debatecited by 384
  2. 2017Safe Model-based Reinforcement Learning with Stability Guaranteescited by 347
  3. 2020Responsive Safety in Reinforcement Learning by PID Lagrangian Methodscited by 302
  4. 2018Safe Exploration in Continuous Action Spacescited by 237
  5. 2018Supervising strong learners by amplifying weak expertscited by 139
  6. 2017Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spacescited by 122
  7. 2020Reward-rational (implicit) choice: A unifying formalism for reward learningcited by 119
  8. 2020Learning to be Safe: Deep RL with a Safety Criticcited by 109
  9. 2019Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspectivecited by 107
  10. 2017Trial without Error: Towards Safe Reinforcement Learning via Human Interventioncited by 102
  11. 2020Offline Learning from Demonstrations and Unlabeled Experiencecited by 72
  12. 2017Recurrent Environment Simulatorscited by 68
  13. 2019Shaping Belief States with Generative Environment Models for RLcited by 65
  14. 2018Learning and Querying Fast Generative Models for Reinforcement Learningcited by 48
  15. 2014Safe Exploration of State and Action Spaces in Reinforcement Learningcited by 37
  16. 2021Learning Barrier Certificates: Towards Safe Reinforcement Learning with Zero Training-time Violationscited by 36
  17. 2021VisuoSpatial Foresight for Physical Sequential Fabric Manipulationcited by 18
  18. 2020REALab: An Embedded Perspective on Tamperingcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 3; 2024: 0; 2025: 0; 2026: 1.

5 papers.

  1. 2023A Survey of Reinforcement Learning from Human Feedbackcited by 259
  2. 2023Architext: Language-Driven Generative Architecture Designcited by 5
  3. 2022Reward Learning with Trees: Methods and Evaluationcited by 2
  4. 2023Learning Interpretable Models of Aircraft Handling Behaviour by Reinforcement Learning from Human Feedbackcited by 0
  5. 2026Learning Safe Agent Behaviour from Human Preferences and Justifications via World Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2017Explore, Exploit or Listen: Combining Human Feedback and Policy Model to Speed up Deep Reinforcement Learning in 3D Worlds
  2. 2026A Scoping Review and Experimental Study on Reinforcement Learning from Human Feedback for Human-Robot Collaboration
  3. 2025PRISM: Projection-based Reward Integration for Scene-Aware Real-to-Sim-to-Real Transfer with Few Demonstrations
  4. 2024Trustworthy Human-AI Collaboration: Reinforcement Learning with Human Feedback and Physics Knowledge for Safe Autonomous Driving
  5. 2026SafeDojo: Safe Reinforcement Learning for VLA via Interactive World Model
  6. 2018DQN-TAMER: Human-in-the-Loop Reinforcement Learning with Intractable Feedback
  7. 2025AGENTSAFE: Benchmarking the Safety of Embodied Agents on Hazardous Instructions
  8. 2023Safe Reinforcement Learning in a Simulated Robotic Arm