# Developing Driving Strategies Efficiently: A Skill-Based Hierarchical Reinforcement Learning Approach

arXiv: https://arxiv.org/abs/2302.02179 · HTML: https://mlsanity.com/paper/2302.02179/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for driving, one of 9 directions in End-to-end autonomous driving.
- Builds on 22 of its 36 references that are in the map.
- First posted on arXiv 4 Feb 2023.

## Placed in

- End-to-end autonomous driving / [Reinforcement learning for driving](https://mlsanity.com/driving/reinforcement-learning-for-driving/)

## Builds on (most cited first, 22 of 22)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [Diversity is All You Need: Learning Skills without a Reward Function](https://mlsanity.com/paper/1802.06070/) (2018)
- [Deep Reinforcement Learning for Autonomous Driving: A Survey](https://mlsanity.com/paper/2002.00444/) (2020)
- [The Option-Critic Architecture](https://mlsanity.com/paper/1609.05140/) (2016)
- [FeUdal Networks for Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1703.01161/) (2017)
- [Dynamics-Aware Unsupervised Discovery of Skills](https://mlsanity.com/paper/1907.01657/) (2019)
- [Deep Reinforcement Learning framework for Autonomous Driving](https://mlsanity.com/paper/1704.02532/) (2017)
- [Efficient Exploration via State Marginal Matching](https://mlsanity.com/paper/1906.05274/) (2019)
- [Reinforcement Learning with Prototypical Representations](https://mlsanity.com/paper/2102.11271/) (2021)
- [Behavior From the Void: Unsupervised Active Pre-Training](https://mlsanity.com/paper/2103.04551/) (2021)
- [Model-free Deep Reinforcement Learning for Urban Autonomous Driving](https://mlsanity.com/paper/1904.09503/) (2019)
- [Kickstarting Deep Reinforcement Learning](https://mlsanity.com/paper/1803.03835/) (2018)
- [A Reinforcement Learning Based Approach for Automated Lane Change Maneuvers](https://mlsanity.com/paper/1804.07871/) (2018)
- [Urban Driving with Multi-Objective Deep Reinforcement Learning](https://mlsanity.com/paper/1811.08586/) (2018)
- [Trajectory Planning for Autonomous Vehicles Using Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2011.04752/) (2020)
- [Subgoal-based Reward Shaping to Improve Efficiency in Reinforcement Learning](https://mlsanity.com/paper/2104.06411/) (2021)
- [Hierarchical Reinforcement Learning for Self-Driving Decision-Making without Reliance on Labeled Driving Data](https://mlsanity.com/paper/2001.09816/) (2020)
- [Designing Rewards for Fast Learning](https://mlsanity.com/paper/2205.15400/) (2022)
- [Hierarchical Kickstarting for Skill Transfer in Reinforcement Learning](https://mlsanity.com/paper/2207.11584/) (2022)

## Built on by (most cited first, 3 of 3)

- [Hybrid Action Based Reinforcement Learning for Multi-Objective Compatible Autonomous Driving](https://mlsanity.com/paper/2501.08096/) (2025)
- [Multi-Timescale Hierarchical Reinforcement Learning for Unified Behavior and Control of Autonomous Driving](https://mlsanity.com/paper/2506.23771/) (2025)
- [Learning to Drive Safely with Hybrid Options](https://mlsanity.com/paper/2510.24674/) (2025)

## Abstract (arXiv)

Driving in dense traffic with human and autonomous drivers is a challenging task that requires high-level planning and reasoning. Human drivers can achieve this task comfortably, and there has been many efforts to model human driver strategies. These strategies can be used as inspirations for developing autonomous driving algorithms or to create high-fidelity simulators. Reinforcement learning is a common tool to model driver policies, but conventional training of these models can be computationally expensive and time-consuming. To address this issue, in this paper, we propose ``skill-based" hierarchical driving strategies, where motion primitives, i.e. skills, are designed and used as high-level actions. This reduces the training time for applications that require multiple models with varying behavior. Simulation results in a merging scenario demonstrate that the proposed approach yields driver models that achieve higher performance with less training compared to baseline reinforcement learning methods.