# From Learning to Mastery: Achieving Safe and Efficient Real-World Autonomous Driving with Human-In-The-Loop Reinforcement Learning

arXiv: https://arxiv.org/abs/2510.06038 · HTML: https://mlsanity.com/paper/2510.06038/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for driving, one of 9 directions in End-to-end autonomous driving.
- Builds on 25 of its 37 references that are in the map.
- First posted on arXiv 7 Oct 2025.

## Placed in

- End-to-end autonomous driving / [Reinforcement learning for driving](https://mlsanity.com/driving/reinforcement-learning-for-driving/)
- Deep reinforcement learning methods / [RL for autonomous driving and traffic](https://mlsanity.com/deeprl/rl-for-autonomous-driving-and-traffic/)

## Builds on (most cited first, 25 of 25)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [Deep Reinforcement Learning for Autonomous Driving: A Survey](https://mlsanity.com/paper/2002.00444/) (2020)
- [MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning](https://mlsanity.com/paper/2109.12674/) (2021)
- [HG-DAgger: Interactive Imitation Learning with Human Experts](https://mlsanity.com/paper/1810.02890/) (2018)
- [Human-in-the-Loop Imitation Learning using Remote Teleoperation](https://mlsanity.com/paper/2012.06733/) (2020)
- [Survey of Deep Reinforcement Learning for Motion Planning of Autonomous Vehicles](https://mlsanity.com/paper/2001.11231/) (2020)
- [ThriftyDAgger: Budget-Aware Novelty and Risk Gating for Interactive Imitation Learning](https://mlsanity.com/paper/2109.08273/) (2021)
- [Reward (Mis)design for Autonomous Driving](https://mlsanity.com/paper/2104.13906/) (2021)
- [EnsembleDAgger: A Bayesian Approach to Safe Imitation Learning](https://mlsanity.com/paper/1807.08364/) (2018)
- [Deep Reinforcement Learning for Intelligent Transportation Systems: A Survey](https://mlsanity.com/paper/2005.00935/) (2020)
- [CIRL: Controllable Imitative Reinforcement Learning for Vision-based Self-driving](https://mlsanity.com/paper/1807.03776/) (2018)
- [Batch Active Preference-Based Learning of Reward Functions](https://mlsanity.com/paper/1810.04303/) (2018)
- [Distributional Soft Actor-Critic: Off-Policy Reinforcement Learning for Addressing Value Estimation Errors](https://mlsanity.com/paper/2001.02811/) (2020)
- [Query-Efficient Imitation Learning for End-to-End Autonomous Driving](https://mlsanity.com/paper/1605.06450/) (2016)
- [Deep Reinforcement Learning for Industrial Insertion Tasks with Visual Inputs and Natural Rewards](https://mlsanity.com/paper/1906.05841/) (2019)
- [Learning Reward Functions by Integrating Human Demonstrations and Preferences](https://mlsanity.com/paper/1906.08928/) (2019)
- [Imitation Bootstrapped Reinforcement Learning](https://mlsanity.com/paper/2311.02198/) (2023)
- [A Survey of Deep RL and IL for Autonomous Driving Policy Learning](https://mlsanity.com/paper/2101.01993/) (2021)
- [Efficient Learning of Safe Driving Policy via Human-AI Copilot Optimization](https://mlsanity.com/paper/2202.10341/) (2022)
- [Distributional Soft Actor-Critic with Three Refinements](https://mlsanity.com/paper/2310.05858/) (2023)
- [Learning from Active Human Involvement through Proxy Value Propagation](https://mlsanity.com/paper/2502.03369/) (2025)

## Built on by (most cited first, 2 of 2)

- [ParkingWorld: End-to-End Autonomous Parking Reinforcement Learning from Corrective Experience in 3DGS Simulation](https://mlsanity.com/paper/2605.25029/) (2026)
- [Beyond Safe Execution: Safe Online Reinforcement Learning for Autonomous Driving with Value Calibration and Adaptive Guidance](https://mlsanity.com/paper/2610.17799/) (2026)

## Abstract (arXiv)

Autonomous driving with reinforcement learning (RL) has significant potential. However, applying RL in real-world settings remains challenging due to the need for safe, efficient, and robust learning. Incorporating human expertise into the learning process can help overcome these challenges by reducing risky exploration and improving sample efficiency. In this work, we propose a reward-free, active human-in-the-loop learning method called Human-Guided Distributional Soft Actor-Critic (H-DSAC). Our method combines Proxy Value Propagation (PVP) and Distributional Soft Actor-Critic (DSAC) to enable efficient and safe training in real-world environments. The key innovation is the construction of a distributed proxy value function within the DSAC framework. This function encodes human intent by assigning higher expected returns to expert demonstrations and penalizing actions that require human intervention. By extrapolating these labels to unlabeled states, the policy is effectively guided toward expert-like behavior. With a well-designed state space, our method achieves real-world driving policy learning within practical training times. Results from both simulation and real-world experiments demonstrate that our framework enables safe, robust, and sample-efficient learning for autonomous driving.