mlsanityData to 7 Oct 2026

Efficient Learning of Safe Driving Policy via Human-AI Copilot Optimization

arXiv 2202.10341 ↗First posted 17 Feb 2022cs.LG cs.AI cs.ROcited by 44

On the map: End-to-end autonomous driving → Reinforcement learning for driving

Abstract (arXiv)

Human intervention is an effective way to inject human knowledge into the training loop of reinforcement learning, which can bring fast learning and ensured training safety. Given the very limited budget of human intervention, it remains challenging to design when and how human expert interacts with the learning agent in the training. In this work, we develop a novel human-in-the-loop learning method called Human-AI Copilot Optimization (HACO).To allow the agent's sufficient exploration in the risky environments while ensuring the training safety, the human expert can take over the control and demonstrate how to avoid probably dangerous situations or trivial behaviors. The proposed HACO then effectively utilizes the data both from the trial-and-error exploration and human's partial demonstration to train a high-performing agent. HACO extracts proxy state-action values from partial human demonstration and optimizes the agent to improve the proxy values meanwhile reduce the human interventions. The experiments show that HACO achieves a substantially high sample efficiency in the safe driving benchmark. HACO can train agents to drive in unseen traffic scenarios with a handful of human intervention budget and achieve high safety and generalizability, outperforming both reinforcement learning and imitation learning baselines with a large margin. Code and demo videos are available at: https://decisionforce.github.io/HACO/.

Key facts. Cited by 44 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for driving, one of 9 directions in End-to-end autonomous driving. Builds on 38 of its 58 references that are in the map. First posted on arXiv 17 Feb 2022.

Builds on

38 of 58 references.

Closest ancestors · 3

Same direction, newest first

  1. 2021Safe Driving via Expert Guided Policy Optimization13 Oct
  2. 2021MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning26 Sep
  3. 2021Human-in-the-Loop Deep Reinforcement Learning with Application to Autonomous Driving15 Apr

Other references · 35

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2017Deep reinforcement learning from human preferencescited by 3,846
  4. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  5. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  6. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  7. 2016Concrete Problems in AI Safetycited by 1,582
  8. 2016Generative Adversarial Imitation Learningcited by 1,576
  9. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  10. 2016End to End Learning for Self-Driving Carscited by 1,000
  11. 2017Constrained Policy Optimizationcited by 928
  12. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
23 more
  1. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  2. 2018An Algorithmic Perspective on Imitation Learningcited by 455
  3. 2017Safe Reinforcement Learning via Shieldingcited by 427
  4. 2018Reward learning from human preferences and demonstrations in Ataricited by 314
  5. 2020Responsive Safety in Reinforcement Learning by PID Lagrangian Methodscited by 302
  6. 2018HG-DAgger: Interactive Imitation Learning with Human Expertscited by 245
  7. 2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learningcited by 203
  8. 2017Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spacescited by 122
  9. 2020Conservative Safety Critics for Explorationcited by 107
  10. 2020Learning to Walk in the Real World with Minimal Human Effortcited by 103
  11. 2020Human-in-the-Loop Imitation Learning using Remote Teleoperationcited by 103
  12. 2017Trial without Error: Towards Safe Reinforcement Learning via Human Interventioncited by 102
  13. 2018Multi-Agent Generative Adversarial Imitation Learningcited by 99
  14. 2021ThriftyDAgger: Budget-Aware Novelty and Risk Gating for Interactive Imitation Learningcited by 98
  15. 2016Query-Efficient Imitation Learning for End-to-End Autonomous Drivingcited by 83
  16. 2018Shared Autonomy via Deep Reinforcement Learningcited by 65
  17. 2019Learning Reward Functions by Integrating Human Demonstrations and Preferencescited by 64
  18. 2019Deep Reinforcement Learning from Policy-Dependent Human Feedbackcited by 56
  19. 2020Widening the Pipeline in Human-Guided Reinforcement Learning with Explanation and Context-Aware Data Augmentationcited by 42
  20. 2020Continual Learning of Control Primitives: Skill Discovery via Reset-Gamescited by 29
  21. 2017Agent-Agnostic Human-in-the-Loop Reinforcement Learningcited by 23
  22. 2019Learning a Decision Module by Imitating Driver's Control Behaviorscited by 17
  23. 2021Recent Advances in Leveraging Human Guidance for Sequential Decision-Making Taskscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 10; 2024: 11; 2025: 14; 2026: 6.

44 papers.

Sort:

By area:

  1. 2023A Survey of Imitation Learning: Algorithms, Recent Developments, and Challengescited by 145
  2. 2023Recent Advancements in End-to-End Autonomous Driving using Deep Learning: A Surveycited by 101
  3. 2024RAG-Driver: Generalisable Driving Explanations with Retrieval-Augmented In-Context Learning in Multi-Modal Large Language Modelcited by 91
  4. 2024Multi-Task Interactive Robot Fleet Learning with Visual World Modelscited by 33
  5. 2024HAIM-DRL: Enhanced Human-in-the-loop Reinforcement Learning for Safe and Efficient Autonomous Drivingcited by 31
  6. 2024Multi-Agent Reinforcement Learning for Autonomous Driving: A Surveycited by 30
  7. 2023Human-in-the-Loop Task and Motion Planning for Imitation Learningcited by 28
  8. 2023RLIF: Interactive Imitation Learning as Reinforcement Learningcited by 24
  9. 2025RoboCopilot: Human-in-the-loop Interactive Imitation Learning for Robot Manipulationcited by 22
  10. 2025Learning from Active Human Involvement through Proxy Value Propagationcited by 16
  11. 2025Human-assisted Robotic Policy Refinement via Action Preference Optimizationcited by 13
  12. 2023Guarded Policy Optimization with Imperfect Online Demonstrationscited by 10
32 more
  1. 2024Weakly-supervised Medical Image Segmentation with Gaze Annotationscited by 10
  2. 2024A Comprehensive Survey on Inverse Constrained Reinforcement Learning: Definitions, Progress and Challengescited by 10
  3. 2023How To Guide Your Learner: Imitation Learning with Active Adaptive Expert Involvementcited by 8
  4. 2023Uncertainty-Aware Decision Transformer for Stochastic Driving Environmentscited by 8
  5. 2024Trustworthy Human-AI Collaboration: Reinforcement Learning with Human Feedback and Physics Knowledge for Safe Autonomous Drivingcited by 8
  6. 2026DeMaVLA: A Vision-Language-Action Foundation Model for Generalizable Deformable Manipulationcited by 8
  7. 2025MILE: Model-based Intervention Learningcited by 7
  8. 2022SafeRL-Kit: Evaluating Efficient Reinforcement Learning Methods for Safe Autonomous Drivingcited by 6
  9. 2025Robot-Gated Interactive Imitation Learning with Adaptive Intervention Mechanismcited by 6
  10. 2024Knowledge Transfer from Simple to Complex: A Safe and Efficient Reinforcement Learning Framework for Autonomous Driving Decision-Makingcited by 5
  11. 2022Adaptive User-Centered Multimodal Interaction towards Reliable and Trusted Automotive Interfacescited by 4
  12. 2024Safe and Robust Reinforcement Learning: Principles and Practicecited by 4
  13. 2024Shared Autonomy with IDA: Interventional Diffusion Assistancecited by 3
  14. 2025Data-Efficient Learning from Human Interventions for Mobile Robotscited by 3
  15. 2025Predictive Preference Learning from Human Interventionscited by 3
  16. 2022Exploring with Sticky Mittens: Reinforcement Learning with Expert Interventions via Option Templatescited by 2
  17. 2024Automated Driving with Evolution Capability: A Reinforcement Learning Method with Monotonic Performance Enhancementcited by 2
  18. 2025AEGIS: Human Attention-based Explainable Guidance for Intelligent Vehicle Systemscited by 2
  19. 2025Sharpening the Spear: Adaptive Expert-Guided Adversarial Attack Against DRL-based Autonomous Driving Policiescited by 2
  20. 2023Teach Me How to Learn: A Perspective Review towards User-centered Neuro-symbolic Learning for Robotic Surgical Systemscited by 1
  21. 2025Robust Driving Control for Autonomous Vehicles: An Intelligent General-sum Constrained Adversarial Reinforcement Learning Approachcited by 1
  22. 2026Bounded Autonomy for Enterprise AI: Typed Action Contracts and Consumer-Side Executioncited by 1
  23. 2023Adaptive User-centered Neuro-symbolic Learning for Multimodal Interaction with Autonomous Systemscited by 0
  24. 2023Multiagent Copilot Approach for Shared Autonomy between Human EEG and TD3 Deep Reinforcement Learningcited by 0
  25. 2025Interactive Double Deep Q-network: Integrating Human Interventions and Evaluative Predictions in Reinforcement Learning of Autonomous Drivingcited by 0
  26. 2025Confidence-Guided Human-AI Collaboration: Reinforcement Learning with Distributional Proxy Value Propagation for Autonomous Drivingcited by 0
  27. 2025From Learning to Mastery: Achieving Safe and Efficient Real-World Autonomous Driving with Human-In-The-Loop Reinforcement Learningcited by 0
  28. 2025End-to-end Autonomous Vehicle Following System using Monocular Fisheye Cameracited by 0
  29. 2026Risk-Aware Human-in-the-Loop Framework with Adaptive Intrusion Response for Autonomous Vehiclescited by 0
  30. 2026CDCP: Conditional Diffusion Model with Contextual Prompts for Multi-task Offline Safe Reinforcement Learningcited by 0
  31. 2026No Free Checker: A Survey of Verifiers for Robot Policiescited by 0
  32. 2026Learning from Runtime Feedback through Failure-Bank Self-Evolution for Vision-Language-Action Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Exploring the trade off between human driving imitation and safety for traffic simulation
  2. 2024Optimizing Autonomous Driving for Safety: A Human-Centric Approach with LLM-Enhanced RLHF
  3. 2023Developing Driving Strategies Efficiently: A Skill-Based Hierarchical Reinforcement Learning Approach
  4. 2024Human-compatible driving partners through data-regularized self-play reinforcement learning
  5. 2026Learning from Human Driving: A Human-in-the-Loop Online Behavior Cloning Framework for Autonomous Driving
  6. 2022Learning from Demonstrations of Critical Driving Behaviours Using Driver's Risk Field
  7. 2024Autonomous Algorithm for Training Autonomous Vehicles with Minimal Human Intervention
  8. 2022Evaluation of MPC-based Imitation Learning for Human-like Autonomous Driving