# Knowledge Transfer from Simple to Complex: A Safe and Efficient Reinforcement Learning Framework for Autonomous Driving Decision-Making

arXiv: https://arxiv.org/abs/2410.14468 · HTML: https://mlsanity.com/paper/2410.14468/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for driving, one of 9 directions in End-to-end autonomous driving.
- Builds on 32 of its 73 references that are in the map.
- First posted on arXiv 18 Oct 2024.

## Placed in

- End-to-end autonomous driving / [Reinforcement learning for driving](https://mlsanity.com/driving/reinforcement-learning-for-driving/)
- Deep reinforcement learning methods / [RL for autonomous driving and traffic](https://mlsanity.com/deeprl/rl-for-autonomous-driving-and-traffic/)

## Builds on (most cited first, 30 of 32)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [End to End Learning for Self-Driving Cars](https://mlsanity.com/paper/1604.07316/) (2016)
- [Responsive Safety in Reinforcement Learning by PID Lagrangian Methods](https://mlsanity.com/paper/2007.03964/) (2020)
- [A Lyapunov-based Approach to Safe Reinforcement Learning](https://mlsanity.com/paper/1805.07708/) (2018)
- [HG-DAgger: Interactive Imitation Learning with Human Experts](https://mlsanity.com/paper/1810.02890/) (2018)
- [Deep Reinforcement Learning framework for Autonomous Driving](https://mlsanity.com/paper/1704.02532/) (2017)
- [Mastering Complex Control in MOBA Games with Deep Reinforcement Learning](https://mlsanity.com/paper/1912.09729/) (2019)
- [A Survey on Negative Transfer](https://mlsanity.com/paper/2009.00909/) (2020)
- [Learning to Walk in the Real World with Minimal Human Effort](https://mlsanity.com/paper/2002.08550/) (2020)
- [Playing FPS Games with Deep Reinforcement Learning](https://mlsanity.com/paper/1609.05521/) (2016)
- [Urban Driving with Conditional Imitation Learning](https://mlsanity.com/paper/1912.00177/) (2019)
- [Constrained Variational Policy Optimization for Safe Reinforcement Learning](https://mlsanity.com/paper/2201.11927/) (2022)
- [A Survey of Deep RL and IL for Autonomous Driving Policy Learning](https://mlsanity.com/paper/2101.01993/) (2021)
- [Efficient Learning of Safe Driving Policy via Human-AI Copilot Optimization](https://mlsanity.com/paper/2202.10341/) (2022)
- [Autonomous Overtaking in Gran Turismo Sport Using Curriculum Reinforcement Learning](https://mlsanity.com/paper/2103.14666/) (2021)
- [HAIM-DRL: Enhanced Human-in-the-loop Reinforcement Learning for Safe and Efficient Autonomous Driving](https://mlsanity.com/paper/2401.03160/) (2024)
- [Safe Driving via Expert Guided Policy Optimization](https://mlsanity.com/paper/2110.06831/) (2021)
- [Deep Reinforcement Learning for Smart Home Energy Management](https://mlsanity.com/paper/1909.10165/) (2019)
- [Game-Theoretic Modeling of Driver and Vehicle Interactions for Verification and Validation of Autonomous Vehicle Control Systems](https://mlsanity.com/paper/1608.08589/) (2016)
- [Model-based Chance-Constrained Reinforcement Learning via Separated Proportional-Integral Lagrangian](https://mlsanity.com/paper/2108.11623/) (2021)
- [Brain Inspired Cognitive Model with Attention for Self-Driving Cars](https://mlsanity.com/paper/1702.05596/) (2017)
- [Enhanced Transfer Learning for Autonomous Driving with Systematic Accident Simulation](https://mlsanity.com/paper/2007.12148/) (2020)
- [Reinforcement Learning from Simulation to Real World Autonomous Driving using Digital Twin](https://mlsanity.com/paper/2211.14874/) (2022)
- [Guarded Policy Optimization with Imperfect Online Demonstrations](https://mlsanity.com/paper/2303.01728/) (2023)
- [Digital Twins for Autonomous Driving: A Comprehensive Implementation and Demonstration](https://mlsanity.com/paper/2401.08653/) (2023)

## Built on by (most cited first, 6 of 6)

- [A Survey of Reinforcement Learning-Based Motion Planning for Autonomous Driving: Lessons Learned from a Driving Task Perspective](https://mlsanity.com/paper/2503.23650/) (2025)
- [Driving Beyond Privilege: Distilling Dense-Reward Knowledge into Sparse-Reward Policies](https://mlsanity.com/paper/2512.04279/) (2025)
- [Neuro-symbolic Action Masking for Deep Reinforcement Learning](https://mlsanity.com/paper/2602.10598/) (2026)
- [TRUST-UP: Trustworthy Reinforcement learning Using Safe Techniques for UAV Pursuit](https://mlsanity.com/paper/2411.17552/) (2024)
- [Mitigating Traffic Oscillations in Mixed Traffic Flow with Scalable Deep Koopman Predictive Control](https://mlsanity.com/paper/2502.00043/) (2025)
- [Evaluating Fuzz Testing for Reinforcement Learning Agents](https://mlsanity.com/paper/2607.24577/) (2026)

## Abstract (arXiv)

A safe and efficient decision-making system is crucial for autonomous vehicles. However, the complexity of driving environments limits the effectiveness of many rule-based and machine learning approaches. Reinforcement Learning (RL), with its robust self-learning capabilities and environmental adaptability, offers a promising solution to these challenges. Nevertheless, safety and efficiency concerns during training hinder its widespread application. To address these concerns, we propose a novel RL framework, Simple to Complex Collaborative Decision (S2CD). First, we rapidly train the teacher model in a lightweight simulation environment. In the more complex and realistic environment, teacher intervenes when the student agent exhibits suboptimal behavior by assessing actions' value to avert dangers. We also introduce an RL algorithm called Adaptive Clipping Proximal Policy Optimization Plus, which combines samples from both teacher and student policies and employs dynamic clipping strategies based on sample importance. This approach improves sample efficiency while effectively alleviating data imbalance. Additionally, we employ the Kullback-Leibler divergence as a policy constraint, transforming it into an unconstrained problem with the Lagrangian method to accelerate the student's learning. Finally, a gradual weaning strategy ensures that the student learns to explore independently over time, overcoming the teacher's limitations and maximizing performance. Simulation experiments in highway lane-change scenarios show that the S2CD framework enhances learning efficiency, reduces training costs, and significantly improves safety compared to state-of-the-art algorithms. This framework also ensures effective knowledge transfer between teacher and student models, even with suboptimal teachers, the student achieves superior performance, demonstrating the robustness and effectiveness of S2CD.