# Extensive Exploration in Complex Traffic Scenarios using Hierarchical Reinforcement Learning

arXiv: https://arxiv.org/abs/2501.14992 · HTML: https://mlsanity.com/paper/2501.14992/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in RL for autonomous driving and traffic, one of 15 directions in Deep reinforcement learning methods.
- Builds on 18 of its 45 references that are in the map.
- First posted on arXiv 25 Jan 2025.

## Placed in

- Deep reinforcement learning methods / [RL for autonomous driving and traffic](https://mlsanity.com/deeprl/rl-for-autonomous-driving-and-traffic/)

## Builds on (most cited first, 18 of 18)

- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [End to End Learning for Self-Driving Cars](https://mlsanity.com/paper/1604.07316/) (2016)
- [Deep Reinforcement Learning for Autonomous Driving: A Survey](https://mlsanity.com/paper/2002.00444/) (2020)
- [Offline Reinforcement Learning as One Big Sequence Modeling Problem](https://mlsanity.com/paper/2106.02039/) (2021)
- [End-to-end Driving via Conditional Imitation Learning](https://mlsanity.com/paper/1710.02410/) (2017)
- [A Survey of Autonomous Driving: Common Practices and Emerging Technologies](https://mlsanity.com/paper/1906.05113/) (2019)
- [Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation](https://mlsanity.com/paper/1604.06057/) (2016)
- [Learning to Drive in a Day](https://mlsanity.com/paper/1807.00412/) (2018)
- [Urban Driving with Conditional Imitation Learning](https://mlsanity.com/paper/1912.00177/) (2019)
- [Explaining How a Deep Neural Network Trained with End-to-End Learning Steers a Car](https://mlsanity.com/paper/1704.07911/) (2017)
- [A Reinforcement Learning Based Approach for Automated Lane Change Maneuvers](https://mlsanity.com/paper/1804.07871/) (2018)
- [Learning hierarchical behavior and motion planning for autonomous driving](https://mlsanity.com/paper/2005.03863/) (2020)
- [Driving Decision and Control for Autonomous Lane Change based on Deep Reinforcement Learning](https://mlsanity.com/paper/1904.10171/) (2019)
- [Driving Intention Recognition and Lane Change Prediction on the Highway](https://mlsanity.com/paper/1908.10820/) (2019)
- [Integrating Deep Reinforcement Learning with Model-based Path Planners for Automated Driving](https://mlsanity.com/paper/2002.00434/) (2020)
- [A Hierarchical Architecture for Sequential Decision-Making in Autonomous Driving using Deep Reinforcement Learning](https://mlsanity.com/paper/1906.08464/) (2019)
- [Multi-lane Cruising Using Hierarchical Planning and Reinforcement Learning](https://mlsanity.com/paper/2110.00650/) (2021)

## Built on by (most cited first, 3 of 3)

- [Multi-Timescale Hierarchical Reinforcement Learning for Unified Behavior and Control of Autonomous Driving](https://mlsanity.com/paper/2506.23771/) (2025)
- [Bootstrapping Reinforcement Learning with Sub-optimal Policies for Autonomous Driving](https://mlsanity.com/paper/2509.04712/) (2025)
- [An Uncertainty-Weighted Decision Transformer for Navigation in Dense, Complex Driving Scenarios](https://mlsanity.com/paper/2509.13132/) (2025)

## Abstract (arXiv)

Developing an automated driving system capable of navigating complex traffic environments remains a formidable challenge. Unlike rule-based or supervised learning-based methods, Deep Reinforcement Learning (DRL) based controllers eliminate the need for domain-specific knowledge and datasets, thus providing adaptability to various scenarios. Nonetheless, a common limitation of existing studies on DRL-based controllers is their focus on driving scenarios with simple traffic patterns, which hinders their capability to effectively handle complex driving environments with delayed, long-term rewards, thus compromising the generalizability of their findings. In response to these limitations, our research introduces a pioneering hierarchical framework that efficiently decomposes intricate decision-making problems into manageable and interpretable subtasks. We adopt a two step training process that trains the high-level controller and low-level controller separately. The high-level controller exhibits an enhanced exploration potential with long-term delayed rewards, and the low-level controller provides longitudinal and lateral control ability using short-term instantaneous rewards. Through simulation experiments, we demonstrate the superiority of our hierarchical controller in managing complex highway driving situations.