# A Safe Hierarchical Planning Framework for Complex Driving Scenarios based on Reinforcement Learning

arXiv: https://arxiv.org/abs/2101.06778 · HTML: https://mlsanity.com/paper/2101.06778/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for driving, one of 9 directions in End-to-end autonomous driving.
- Builds on 18 of its 28 references that are in the map.
- First posted on arXiv 17 Jan 2021.

## Placed in

- End-to-end autonomous driving / [Reinforcement learning for driving](https://mlsanity.com/driving/reinforcement-learning-for-driving/)

## Builds on (most cited first, 18 of 18)

- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [End to End Learning for Self-Driving Cars](https://mlsanity.com/paper/1604.07316/) (2016)
- [A Survey of Motion Planning and Control Techniques for Self-driving Urban Vehicles](https://mlsanity.com/paper/1604.07446/) (2016)
- [INTERACTION Dataset: An INTERnational, Adversarial and Cooperative moTION Dataset in Interactive Driving Scenarios with Semantic Maps](https://mlsanity.com/paper/1910.03088/) (2019)
- [Deep Reinforcement Learning framework for Autonomous Driving](https://mlsanity.com/paper/1704.02532/) (2017)
- [End-to-end Learning of Driving Models from Large-scale Video Datasets](https://mlsanity.com/paper/1612.01079/) (2016)
- [Interpretable End-to-end Urban Autonomous Driving with Latent Deep Reinforcement Learning](https://mlsanity.com/paper/2001.08726/) (2020)
- [Virtual to Real Reinforcement Learning for Autonomous Driving](https://mlsanity.com/paper/1704.03952/) (2017)
- [Conditional Generative Neural System for Probabilistic Trajectory Prediction](https://mlsanity.com/paper/1905.01631/) (2019)
- [Combining Planning and Deep Reinforcement Learning in Tactical Decision Making for Autonomous Driving](https://mlsanity.com/paper/1905.02680/) (2019)
- [Deep Imitation Learning for Autonomous Driving in Generic Urban Scenarios with Enhanced Safety](https://mlsanity.com/paper/1903.00640/) (2019)
- [Social Attention for Autonomous Decision-Making in Dense Traffic](https://mlsanity.com/paper/1911.12250/) (2019)
- [Reinforcement Learning based Control of Imitative Policies for Near-Accident Driving](https://mlsanity.com/paper/2007.00178/) (2020)
- [A Fast Integrated Planning and Control Framework for Autonomous Driving via Imitation Learning](https://mlsanity.com/paper/1707.02515/) (2017)
- [Learning hierarchical behavior and motion planning for autonomous driving](https://mlsanity.com/paper/2005.03863/) (2020)
- [Hierarchical Reinforcement Learning Method for Autonomous Vehicle Behavior Planning](https://mlsanity.com/paper/1911.03799/) (2019)
- [Bayesian Persuasive Driving](https://mlsanity.com/paper/1809.09735/) (2018)
- [A Deep Reinforcement Learning Driving Policy for Autonomous Road Vehicles](https://mlsanity.com/paper/1905.09046/) (2019)

## Built on by (most cited first, 18 of 18)

- [Motion Planning for Autonomous Driving: The State of the Art and Future Perspectives](https://mlsanity.com/paper/2303.09824/) (2023)
- [Hierarchical Planning Through Goal-Conditioned Offline Reinforcement Learning](https://mlsanity.com/paper/2205.11790/) (2022)
- [Guided Online Distillation: Promoting Safe Reinforcement Learning by Offline Demonstration](https://mlsanity.com/paper/2309.09408/) (2023)
- [CALMM-Drive: Confidence-Aware Autonomous Driving with Large Multimodal Model](https://mlsanity.com/paper/2412.04209/) (2024)
- [VLN-Zero: Rapid Exploration and Cache-Enabled Neurosymbolic Vision-Language Planning for Zero-Shot Transfer in Robot Navigation](https://mlsanity.com/paper/2509.18592/) (2025)
- [EnsembleFollower: A Hybrid Car-Following Framework Based On Reinforcement Learning and Hierarchical Planning](https://mlsanity.com/paper/2308.16008/) (2023)
- [A Survey of Generative AI for Intelligent Transportation Systems: Road Transportation Perspective](https://mlsanity.com/paper/2312.08248/) (2023)
- [SafePath: Conformal Prediction for Safe LLM-Based Autonomous Navigation](https://mlsanity.com/paper/2505.09427/) (2025)
- [Imagination-Augmented Hierarchical Reinforcement Learning for Safe and Interactive Autonomous Driving in Urban Environments](https://mlsanity.com/paper/2311.10309/) (2023)
- [Bi-Level Optimization Augmented with Conditional Variational Autoencoder for Autonomous Driving in Dense Traffic](https://mlsanity.com/paper/2212.02224/) (2022)
- [End-to-End Learning of Behavioural Inputs for Autonomous Driving in Dense Traffic](https://mlsanity.com/paper/2310.14766/) (2023)
- [A Survey of Reinforcement Learning-Based Motion Planning for Autonomous Driving: Lessons Learned from a Driving Task Perspective](https://mlsanity.com/paper/2503.23650/) (2025)
- [Adaptive Decision Making at the Intersection for Autonomous Vehicles Based on Skill Discovery](https://mlsanity.com/paper/2207.11724/) (2022)
- [Act Better by Timing: A timing-Aware Reinforcement Learning for Autonomous Driving](https://mlsanity.com/paper/2406.13223/) (2024)
- [Let Hybrid A* Path Planner Obey Traffic Rules: A Deep Reinforcement Learning-Based Planning Framework](https://mlsanity.com/paper/2407.01216/) (2024)
- [Decision-Making with Lightweight Confidence-Aware Language Model for Autonomous Driving](https://mlsanity.com/paper/2605.25393/) (2026)
- [Constraint-Enhanced Reinforcement Learning Based on Dynamic Decoupled Spherical Radial Squashing](https://mlsanity.com/paper/2605.04185/) (2026)
- [LUNA-AD: Lightweight Uncertainty-Aware Language Model with Lifelong Learning for Autonomous Driving](https://mlsanity.com/paper/2606.08470/) (2026)

## Abstract (arXiv)

Autonomous vehicles need to handle various traffic conditions and make safe and efficient decisions and maneuvers. However, on the one hand, a single optimization/sampling-based motion planner cannot efficiently generate safe trajectories in real time, particularly when there are many interactive vehicles near by. On the other hand, end-to-end learning methods cannot assure the safety of the outcomes. To address this challenge, we propose a hierarchical behavior planning framework with a set of low-level safe controllers and a high-level reinforcement learning algorithm (H-CtRL) as a coordinator for the low-level controllers. Safety is guaranteed by the low-level optimization/sampling-based controllers, while the high-level reinforcement learning algorithm makes H-CtRL an adaptive and efficient behavior planner. To train and test our proposed algorithm, we built a simulator that can reproduce traffic scenes using real-world datasets. The proposed H-CtRL is proved to be effective in various realistic simulation scenarios, with satisfying performance in terms of both safety and efficiency.