# Behavior Planning at Urban Intersections through Hierarchical Reinforcement Learning

arXiv: https://arxiv.org/abs/2011.04697 · HTML: https://mlsanity.com/paper/2011.04697/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Nov 2020.

## Built on by (most cited first, 12 of 12)

- [Reward-Driven Automated Curriculum Learning for Interaction-Aware Self-Driving at Unsignalized Intersections](https://mlsanity.com/paper/2403.13674/) (2024)
- [Action and Trajectory Planning for Urban Autonomous Driving with Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2306.15968/) (2023)
- [Autonomous Driving at Unsignalized Intersections: A Review of Decision-Making Challenges and Reinforcement Learning-Based Solutions](https://mlsanity.com/paper/2409.13144/) (2024)
- [Advancing Explainable Autonomous Vehicle Systems: A Comprehensive Review and Research Roadmap](https://mlsanity.com/paper/2404.00019/) (2024)
- [Dynamic High-Order Control Barrier Functions with Diffuser for Safety-Critical Trajectory Planning at Signal-Free Intersections](https://mlsanity.com/paper/2412.00162/) (2024)
- [Bilevel Multi-Armed Bandit-Based Hierarchical Reinforcement Learning for Interaction-Aware Self-Driving at Unsignalized Intersections](https://mlsanity.com/paper/2502.03960/) (2025)
- [Imagination-Augmented Hierarchical Reinforcement Learning for Safe and Interactive Autonomous Driving in Urban Environments](https://mlsanity.com/paper/2311.10309/) (2023)
- [A Survey of Reinforcement Learning-Based Motion Planning for Autonomous Driving: Lessons Learned from a Driving Task Perspective](https://mlsanity.com/paper/2503.23650/) (2025)
- [Trajectory Planning for Autonomous Vehicle Using Iterative Reward Prediction in Reinforcement Learning](https://mlsanity.com/paper/2404.12079/) (2024)
- [Let Hybrid A* Path Planner Obey Traffic Rules: A Deep Reinforcement Learning-Based Planning Framework](https://mlsanity.com/paper/2407.01216/) (2024)
- [Beyond Self-Play: Hierarchical Reasoning for Continuous Motion in Closed-Loop Traffic Simulation](https://mlsanity.com/paper/2605.09153/) (2026)
- [Goal-conditioned Hierarchical Reinforcement Learning for Sample-efficient and Safe Autonomous Driving at Intersections](https://mlsanity.com/paper/2506.16336/) (2025)

## Abstract (arXiv)

For autonomous vehicles, effective behavior planning is crucial to ensure safety of the ego car. In many urban scenarios, it is hard to create sufficiently general heuristic rules, especially for challenging scenarios that some new human drivers find difficult. In this work, we propose a behavior planning structure based on reinforcement learning (RL) which is capable of performing autonomous vehicle behavior planning with a hierarchical structure in simulated urban environments. Application of the hierarchical structure allows the various layers of the behavior planning system to be satisfied. Our algorithms can perform better than heuristic-rule-based methods for elective decisions such as when to turn left between vehicles approaching from the opposite direction or possible lane-change when approaching an intersection due to lane blockage or delay in front of the ego car. Such behavior is hard to evaluate as correct or incorrect, but for some aggressive expert human drivers handle such scenarios effectively and quickly. On the other hand, compared to traditional RL methods, our algorithm is more sample-efficient, due to the use of a hybrid reward mechanism and heuristic exploration during the training process. The results also show that the proposed method converges to an optimal policy faster than traditional RL methods.