mlsanityData to 7 Oct 2026

Trial without Error: Towards Safe Reinforcement Learning via Human Intervention

arXiv 1707.05173 ↗First posted 17 Jul 2017cs.AI cs.LG cs.NEcited by 102

Not on the map. Why?

Abstract (arXiv)

AI systems are increasingly applied to complex tasks that involve interaction with humans. During training, such systems are potentially dangerous, as they haven't yet learned to avoid actions that could cause serious harm. How can an AI system explore and learn without making a single mistake that harms humans or otherwise causes serious damage? For model-free reinforcement learning, having a human "in the loop" and ready to intervene is currently the only way to prevent all catastrophes. We formalize human intervention for RL and show how to reduce the human labor required by training a supervised learner to imitate the human's intervention decisions. We evaluate this scheme on Atari games, with a Deep RL agent being overseen by a human for four hours. When the class of catastrophes is simple, we are able to prevent all catastrophes without affecting the agent's learning (whereas an RL baseline fails due to catastrophic forgetting). However, this scheme is less successful when catastrophes are more complex: it reduces but does not eliminate catastrophes and the supervised learner fails on adversarial examples found by the agent. Extrapolating to more challenging environments, we show that our implementation would not scale (due to the infeasible amount of human labor required). We outline extensions of the scheme that are necessary if we are to train model-free agents without a single catastrophe.

Key facts. Cited by 102 papers in the map (data through 7 Oct 2026). First posted on arXiv 17 Jul 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 3; 2019: 0; 2020: 1; 2021: 25; 2022: 16; 2023: 12; 2024: 15; 2025: 15; 2026: 15.

102 papers.

Sort:

By area:

  1. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
  2. 2018Reward learning from human preferences and demonstrations in Ataricited by 314
  3. 2021Unsolved Problems in ML Safetycited by 291
  4. 2018HG-DAgger: Interactive Imitation Learning with Human Expertscited by 245
  5. 2022Exploration in Deep Reinforcement Learning: A Surveycited by 158
  6. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 145
  7. 2022Constrained Variational Policy Optimization for Safe Reinforcement Learningcited by 69
  8. 2021Software Engineering for AI-Based Systems: A Surveycited by 68
  9. 2023Constrained Decision Transformer for Offline Safe Reinforcement Learningcited by 65
  10. 2021Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domaincited by 57
  11. 2024TRANSIC: Sim-to-Real Policy Transfer by Learning from Online Correctioncited by 50
  12. 2022Efficient Learning of Safe Driving Policy via Human-AI Copilot Optimizationcited by 44
38 more
  1. 2022On the Robustness of Safe Reinforcement Learning under Observational Perturbationscited by 38
  2. 2021Direct Behavior Specification via Constrained Reinforcement Learningcited by 30
  3. 2021Safe Driving via Expert Guided Policy Optimizationcited by 29
  4. 2021MORAL: Aligning AI with Human Norms through Multi-Objective Reinforced Active Learningcited by 24
  5. 2022Law Informs Code: A Legal Informatics Approach to Aligning Artificial Intelligence with Humanscited by 23
  6. 2022Provably Safe Reinforcement Learning via Action Projection using Reachability Analysis and Polynomial Zonotopescited by 23
  7. 2024REvolve: Reward Evolution with Large Language Models using Human Feedbackcited by 22
  8. 2021Prioritized Experience-based Reinforcement Learning with Human Guidance for Autonomous Drivingcited by 21
  9. 2023Constraint-Conditioned Policy Optimization for Versatile Safe Reinforcement Learningcited by 17
  10. 2021Human-in-the-Loop Deep Reinforcement Learning with Application to Autonomous Drivingcited by 16
  11. 2025Learning from Active Human Involvement through Proxy Value Propagationcited by 16
  12. 2021Shielding Atari Games with Bounded Presciencecited by 14
  13. 2021Safe Exploration by Solving Early Terminated MDPcited by 13
  14. 2021Recent Advances in Leveraging Human Guidance for Sequential Decision-Making Taskscited by 13
  15. 2023A Human-Centered Safe Robot Reinforcement Learning Framework with Interactive Behaviorscited by 13
  16. 2025A Survey of Safe Reinforcement Learning and Constrained MDPs: A Technical Survey on Single-Agent and Multi-Agent Safetycited by 12
  17. 2021How RL Agents Behave When Their Actions Are Modifiedcited by 10
  18. 2021Training a Resilient Q-Network against Observational Interferencecited by 9
  19. 2021Preference learning along multiple criteria: A game-theoretic perspectivecited by 9
  20. 2024Large Language Model guided Deep Reinforcement Learning for Decision Making in Autonomous Drivingcited by 8
  21. 2021OnSlicing: Online End-to-End Network Slicing with Reinforcement Learningcited by 7
  22. 2022Provable Safe Reinforcement Learning with Binary Feedbackcited by 7
  23. 2025MILE: Model-based Intervention Learningcited by 7
  24. 2020SAFARI: Safe and Active Robot Imitation Learning with Imaginationcited by 5
  25. 2021Provably Correct Training of Neural Network Controllers Using Reachability Analysiscited by 5
  26. 2021Look Before You Leap: Safe Model-Based Reinforcement Learning with Human Interventioncited by 5
  27. 2022Safe Deep RL in 3D Environments using Human Feedbackcited by 5
  28. 2024Safety through feedback in Constrained RLcited by 5
  29. 2026Modeling Distinct Human Interaction in Web Agentscited by 5
  30. 2022Human-AI Shared Control via Policy Dissectioncited by 4
  31. 2024Towards Socially and Morally Aware RL agent: Reward Design With LLMcited by 4
  32. 2024Safe and Robust Reinforcement Learning: Principles and Practicecited by 4
  33. 2024Critic as Lyapunov function (CALF): a model-free, stability-ensuring agentcited by 4
  34. 2021Play to Grade: Testing Coding Games as Classifying Markov Decision Processcited by 3
  35. 2021Risk Sensitive Model-Based Reinforcement Learning using Uncertainty Guided Planningcited by 3
  36. 2021Combining Learning from Human Feedback and Knowledge Engineering to Solve Hierarchical Tasks in Minecraftcited by 3
  37. 2021Cooperation for Scalable Supervision of Autonomy in Mixed Trafficcited by 3
  38. 2021Model-Based Safe Reinforcement Learning with Time-Varying State and Control Constraints: An Application to Intelligent Vehiclescited by 3

50 most cited of 102 All 102 (JSON)