Spatially and Seamlessly Hierarchical Reinforcement Learning for State Space and Policy space in Autonomous Driving
On the map: End-to-end autonomous driving → Reinforcement learning for driving
Abstract (arXiv)
Despite advances in hierarchical reinforcement learning, its applications to path planning in autonomous driving on highways are challenging. One reason is that conventional hierarchical reinforcement learning approaches are not amenable to autonomous driving due to its riskiness: the agent must move avoiding multiple obstacles such as other agents that are highly unpredictable, thus safe regions are small, scattered, and changeable over time. To overcome this challenge, we propose a spatially hierarchical reinforcement learning method for state space and policy space. The high-level policy selects not only behavioral sub-policy but also regions to pay mind to in state space and for outline in policy space. Subsequently, the low-level policy elaborates the short-term goal position of the agent within the outline of the region selected by the high-level command. The network structure and optimization suggested in our method are as concise as those of single-level methods. Experiments on the environment with various shapes of roads showed that our method finds the nearly optimal policies from early episodes, outperforming a baseline hierarchical reinforcement learning method, especially in narrow and complex roads. The resulting trajectories on the roads were similar to those of human strategies on the behavioral planning level.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for driving, one of 9 directions in End-to-end autonomous driving. Builds on 16 of its 32 references that are in the map. First posted on arXiv 10 Nov 2021.
Builds on
16 of 32 references.
References · 16
- 2017Proximal Policy Optimization Algorithmscited by 15,215
- 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
- 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
- 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
- 2018Data-Efficient Hierarchical Reinforcement Learningcited by 455
- 2016The Option-Critic Architecturecited by 448
- 2017FeUdal Networks for Hierarchical Reinforcement Learningcited by 337
- 2016Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivationcited by 286
- 2018Learning to Drive in a Daycited by 273
- 2018Latent Space Policies for Hierarchical Reinforcement Learningcited by 67
- 2019Sub-policy Adaptation for Hierarchical Reinforcement Learningcited by 47
- 2018A Reinforcement Learning Based Approach for Automated Lane Change Maneuverscited by 46
4 more
- 2019Social Attention for Autonomous Decision-Making in Dense Trafficcited by 39
- 2017Combining Neural Networks and Tree Search for Task and Motion Planning in Challenging Environmentscited by 19
- 2017Formulation of Deep Reinforcement Learning Architecture Toward Autonomous Driving for On-Ramp Mergecited by 19
- 2021Multi-lane Cruising Using Hierarchical Planning and Reinforcement Learningcited by 3
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Multi-Timescale Hierarchical Reinforcement Learning for Unified Behavior and Control of Autonomous Driving
- 2023Action and Trajectory Planning for Urban Autonomous Driving with Hierarchical Reinforcement Learning
- 2023Developing Driving Strategies Efficiently: A Skill-Based Hierarchical Reinforcement Learning Approach
- 2025Goal-conditioned Hierarchical Reinforcement Learning for Sample-efficient and Safe Autonomous Driving at Intersections
- 2021A Safe Hierarchical Planning Framework for Complex Driving Scenarios based on Reinforcement Learning
- 2025Extensive Exploration in Complex Traffic Scenarios using Hierarchical Reinforcement Learning
- 2025Learning to Drive Safely with Hybrid Options
- 2020Trajectory Planning for Autonomous Vehicles Using Hierarchical Reinforcement Learning