# Integrating Expert Guidance for Efficient Learning of Safe Overtaking in Autonomous Driving Using Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/2308.09456 · HTML: https://mlsanity.com/paper/2308.09456/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for driving, one of 9 directions in End-to-end autonomous driving.
- Builds on 15 of its 45 references that are in the map.
- First posted on arXiv 18 Aug 2023.

## Placed in

- End-to-end autonomous driving / [Reinforcement learning for driving](https://mlsanity.com/driving/reinforcement-learning-for-driving/)

## Builds on (most cited first, 15 of 15)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Safe Reinforcement Learning via Shielding](https://mlsanity.com/paper/1708.08611/) (2017)
- [Control Barrier Functions: Theory and Applications](https://mlsanity.com/paper/1903.11199/) (2019)
- [End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Tasks](https://mlsanity.com/paper/1903.08792/) (2019)
- [Residual Reinforcement Learning for Robot Control](https://mlsanity.com/paper/1812.03201/) (2018)
- [Residual Policy Learning](https://mlsanity.com/paper/1812.06298/) (2018)
- [Learning Deep Control Policies for Autonomous Aerial Vehicles with MPC-Guided Policy Search](https://mlsanity.com/paper/1509.06791/) (2015)
- [Efficient Reinforcement Learning for Autonomous Driving with Parameterized Skills and Priors](https://mlsanity.com/paper/2305.04412/) (2023)
- [MPC-Net: A First Principles Guided Policy Search](https://mlsanity.com/paper/1909.05197/) (2019)
- [Automated Lane Change Strategy using Proximal Policy Optimization-based Deep Reinforcement Learning](https://mlsanity.com/paper/2002.02667/) (2020)
- [Adaptive Behavior Generation for Autonomous Driving using Deep Reinforcement Learning with Compact Semantic States](https://mlsanity.com/paper/1809.03214/) (2018)
- [Planning for Safe Abortable Overtaking Maneuvers in Autonomous Driving](https://mlsanity.com/paper/2104.00077/) (2021)
- [Differential Dynamic Programming with Nonlinear Safety Constraints Under System Uncertainties](https://mlsanity.com/paper/2011.01051/) (2020)
- [Constrained Trajectory Optimization on Matrix Lie Groups via Lie-Algebraic Differential Dynamic Programming](https://mlsanity.com/paper/2301.02018/) (2023)

## Built on by (most cited first, 1 of 1)

- [A Survey of Reinforcement Learning-Based Motion Planning for Autonomous Driving: Lessons Learned from a Driving Task Perspective](https://mlsanity.com/paper/2503.23650/) (2025)

## Abstract (arXiv)

Overtaking on two-lane roads is a safety-critical decision-making problem for autonomous vehicles, since oncoming traffic may force the ego vehicle to abort and merge back to its original lane. Deep reinforcement learning (DRL) is promising for such continuous-control tasks, but it requires substantial interaction data and may produce unsafe exploratory actions during training. Model-based controllers provide structured behavior, but depend on model fidelity and hand-designed logic. This paper proposes a fading expert-guidance framework that uses a model-based controller to guide DRL training for autonomous overtaking. The expert combines a constrained iterative LQR (CiLQR) planner with PID-based auxiliary controllers, activated when the optimized plan becomes infeasible. The expert action enters the actor objective through a fading term, interpreted as a time-varying soft trust region around the expert policy. This term first biases the policy toward the expert and then vanishes, allowing optimization according to the reinforcement-learning objective. The method is evaluated with PPO, TD3, and SAC. Simulations with bidirectional traffic show improved sample efficiency and final task performance. The safety effects are algorithm-dependent: guidance reduces vehicle collisions for SAC, eliminates boundary collisions for TD3, and leaves the PPO collision profile unchanged. Overall, the results support fading expert guidance as a practical mechanism for transferring model-based driving knowledge to learning-based controllers without restricting the final policy to imitation in an abortable overtaking task with bidirectional traffic.