# Multi-lane Cruising Using Hierarchical Planning and Reinforcement Learning

arXiv: https://arxiv.org/abs/2110.00650 · HTML: https://mlsanity.com/paper/2110.00650/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 17 references that are in the map.
- First posted on arXiv 1 Oct 2021.

## Builds on (most cited first, 7 of 7)

- [End to End Learning for Self-Driving Cars](https://mlsanity.com/paper/1604.07316/) (2016)
- [Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation](https://mlsanity.com/paper/1604.06057/) (2016)
- [A Survey of Motion Planning and Control Techniques for Self-driving Urban Vehicles](https://mlsanity.com/paper/1604.07446/) (2016)
- [Watch This: Scalable Cost-Function Learning for Path Planning in Urban Environments](https://mlsanity.com/paper/1607.02329/) (2016)
- [A Fast Integrated Planning and Control Framework for Autonomous Driving via Imitation Learning](https://mlsanity.com/paper/1707.02515/) (2017)
- [Combining Neural Networks and Tree Search for Task and Motion Planning in Challenging Environments](https://mlsanity.com/paper/1703.07887/) (2017)
- [Long-term Planning by Short-term Prediction](https://mlsanity.com/paper/1602.01580/) (2016)

## Built on by (most cited first, 3 of 3)

- [Extensive Exploration in Complex Traffic Scenarios using Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2501.14992/) (2025)
- [Learning Isometric Embeddings of Road Networks using Multidimensional Scaling](https://mlsanity.com/paper/2504.17534/) (2025)
- [Spatially and Seamlessly Hierarchical Reinforcement Learning for State Space and Policy space in Autonomous Driving](https://mlsanity.com/paper/2111.05479/) (2021)

## Abstract (arXiv)

Competent multi-lane cruising requires using lane changes and within-lane maneuvers to achieve good speed and maintain safety. This paper proposes a design for autonomous multi-lane cruising by combining a hierarchical reinforcement learning framework with a novel state-action space abstraction. While the proposed solution follows the classical hierarchy of behavior decision, motion planning and control, it introduces a key intermediate abstraction within the motion planner to discretize the state-action space according to high level behavioral decisions. We argue that this design allows principled modular extension of motion planning, in contrast to using either monolithic behavior cloning or a large set of hand-written rules. Moreover, we demonstrate that our state-action space abstraction allows transferring of the trained models without retraining from a simulated environment with virtually no dynamics to one with significantly more realistic dynamics. Together, these results suggest that our proposed hierarchical architecture is a promising way to allow reinforcement learning to be applied to complex multi-lane cruising in the real world.