# Human-in-the-Loop Deep Reinforcement Learning with Application to Autonomous Driving

arXiv: https://arxiv.org/abs/2104.07246 · HTML: https://mlsanity.com/paper/2104.07246/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for driving, one of 9 directions in End-to-end autonomous driving.
- Builds on 21 of its 48 references that are in the map.
- First posted on arXiv 15 Apr 2021.

## Placed in

- End-to-end autonomous driving / [Reinforcement learning for driving](https://mlsanity.com/driving/reinforcement-learning-for-driving/)

## Builds on (most cited first, 21 of 21)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations](https://mlsanity.com/paper/1709.10087/) (2017)
- [End-to-end Driving via Conditional Imitation Learning](https://mlsanity.com/paper/1710.02410/) (2017)
- [The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision](https://mlsanity.com/paper/1904.12584/) (2019)
- [Exploring the Limitations of Behavior Cloning for Autonomous Driving](https://mlsanity.com/paper/1904.08980/) (2019)
- [Overcoming Exploration in Reinforcement Learning with Demonstrations](https://mlsanity.com/paper/1709.10089/) (2017)
- [Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards](https://mlsanity.com/paper/1707.08817/) (2017)
- [Reward learning from human preferences and demonstrations in Atari](https://mlsanity.com/paper/1811.06521/) (2018)
- [Deep Reinforcement Learning framework for Autonomous Driving](https://mlsanity.com/paper/1704.02532/) (2017)
- [Never Give Up: Learning Directed Exploration Strategies](https://mlsanity.com/paper/2002.06038/) (2020)
- [Interactive Learning from Policy-Dependent Human Feedback](https://mlsanity.com/paper/1701.06049/) (2017)
- [Count-Based Exploration with the Successor Representation](https://mlsanity.com/paper/1807.11622/) (2018)
- [Trial without Error: Towards Safe Reinforcement Learning via Human Intervention](https://mlsanity.com/paper/1707.05173/) (2017)
- [Hindsight Credit Assignment](https://mlsanity.com/paper/1912.02503/) (2019)
- [A Reinforcement Learning Based Approach for Automated Lane Change Maneuvers](https://mlsanity.com/paper/1804.07871/) (2018)
- [Multi-modal Sensor Fusion-Based Deep Neural Network for End-to-end Autonomous Driving with Scene Understanding](https://mlsanity.com/paper/2005.09202/) (2020)
- [High-speed Autonomous Drifting with Deep Reinforcement Learning](https://mlsanity.com/paper/2001.01377/) (2020)
- [Intervention Aided Reinforcement Learning for Safe and Practical Policy Optimization in Navigation](https://mlsanity.com/paper/1811.06187/) (2018)
- [Neuro-symbolic Architectures for Context Understanding](https://mlsanity.com/paper/2003.04707/) (2020)

## Built on by (most cited first, 16 of 16)

- [Efficient Learning of Safe Driving Policy via Human-AI Copilot Optimization](https://mlsanity.com/paper/2202.10341/) (2022)
- [Multi-Agent Reinforcement Learning for Autonomous Driving: A Survey](https://mlsanity.com/paper/2408.09675/) (2024)
- [Offline Reinforcement Learning for Autonomous Driving with Safety and Exploration Enhancement](https://mlsanity.com/paper/2110.07067/) (2021)
- [Augmenting Reinforcement Learning with Transformer-based Scene Representation Learning for Decision-making of Autonomous Driving](https://mlsanity.com/paper/2208.12263/) (2022)
- [Decision Making for Human-in-the-loop Robotic Agents via Uncertainty-Aware Reinforcement Learning](https://mlsanity.com/paper/2303.06710/) (2023)
- [Knowledge Augmented Complex Problem Solving with Large Language Models: A Survey](https://mlsanity.com/paper/2505.03418/) (2025)
- [A-Eye: Driving with the Eyes of AI for Corner Case Generation](https://mlsanity.com/paper/2202.10803/) (2022)
- [Safe Decision-making for Lane-change of Autonomous Vehicles via Human Demonstration-aided Reinforcement Learning](https://mlsanity.com/paper/2207.00448/) (2022)
- [Human-AI Shared Control via Policy Dissection](https://mlsanity.com/paper/2206.00152/) (2022)
- [Advancing Investment Frontiers: Industry-grade Deep Reinforcement Learning for Portfolio Optimization](https://mlsanity.com/paper/2403.07916/) (2024)
- [Uncertainty-Aware Model-Based Reinforcement Learning with Application to Autonomous Driving](https://mlsanity.com/paper/2106.12194/) (2021)
- [Exploring the trade off between human driving imitation and safety for traffic simulation](https://mlsanity.com/paper/2208.04803/) (2022)
- [Sampling Efficient Deep Reinforcement Learning through Preference-Guided Stochastic Exploration](https://mlsanity.com/paper/2206.09627/) (2022)
- [Interactive Double Deep Q-network: Integrating Human Interventions and Evaluative Predictions in Reinforcement Learning of Autonomous Driving](https://mlsanity.com/paper/2505.01440/) (2025)
- [Bridging Policy and Real-World Dynamics: LLM-Augmented Rebalancing for Shared Micromobility Systems](https://mlsanity.com/paper/2603.00176/) (2026)
- [Themis: An explainable AI-enabled framework for Reinforcement Learning with Human Feedback](https://mlsanity.com/paper/2606.24622/) (2026)

## Abstract (arXiv)

Due to the limited smartness and abilities of machine intelligence, currently autonomous vehicles are still unable to handle all kinds of situations and completely replace drivers. Because humans exhibit strong robustness and adaptability in complex driving scenarios, it is of great importance to introduce humans into the training loop of artificial intelligence, leveraging human intelligence to further advance machine learning algorithms. In this study, a real-time human-guidance-based deep reinforcement learning (Hug-DRL) method is developed for policy training of autonomous driving. Leveraging a newly designed control transfer mechanism between human and automation, human is able to intervene and correct the agent's unreasonable actions in real time when necessary during the model training process. Based on this human-in-the-loop guidance mechanism, an improved actor-critic architecture with modified policy and value networks is developed. The fast convergence of the proposed Hug-DRL allows real-time human guidance actions to be fused into the agent's training loop, further improving the efficiency and performance of deep reinforcement learning. The developed method is validated by human-in-the-loop experiments with 40 subjects and compared with other state-of-the-art learning approaches. The results suggest that the proposed method can effectively enhance the training efficiency and performance of the deep reinforcement learning algorithm under human guidance, without imposing specific requirements on participant expertise and experience.