mlsanityData to 14 Oct 2026

Multi-Timescale Hierarchical Reinforcement Learning for Unified Behavior and Control of Autonomous Driving

arXiv 2506.23771 ↗First posted 30 Jun 2025cs.RO cs.AIcited by 0

On the map: End-to-end autonomous driving → Reinforcement learning for driving

also in 1 more
Abstract (arXiv)

Reinforcement Learning (RL) is increasingly used in autonomous driving (AD) and shows clear advantages. However, most RL-based AD methods overlook policy structure design. An RL policy that only outputs short-timescale vehicle control commands results in fluctuating driving behavior due to fluctuations in network outputs, while one that only outputs long-timescale driving goals cannot achieve unified optimality of driving behavior and control. Therefore, we propose a multi-timescale hierarchical reinforcement learning approach. Our approach adopts a hierarchical policy structure, where high- and low-level RL policies are unified-trained to produce long-timescale motion guidance and short-timescale control commands, respectively. Therein, motion guidance is explicitly represented by hybrid actions to capture multimodal driving behaviors on structured road and support incremental low-level extend-state updates. Additionally, a hierarchical safety mechanism is designed to ensure multi-timescale safety. Evaluation in simulator-based and HighD dataset-based highway multi-lane scenarios demonstrates that our approach significantly improves AD performance, effectively increasing driving efficiency, action consistency and safety.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for driving, one of 9 directions in End-to-end autonomous driving. Builds on 16 of its 38 references that are in the map. First posted on arXiv 30 Jun 2025.

Builds on

16 of 38 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Hybrid Action Based Reinforcement Learning for Multi-Objective Compatible Autonomous Driving14 Jan
  2. 2023Risk-anticipatory autonomous driving strategies considering vehicles' weights, based on hierarchical deep reinforcement learning27 Dec
  3. 2023Parameterized Decision-making with Multi-modal Perception for Autonomous Driving19 Dec
  4. 2023Imagination-Augmented Hierarchical Reinforcement Learning for Safe and Interactive Autonomous Driving in Urban Environments17 Nov
  5. 2023Efficient Reinforcement Learning for Autonomous Driving with Parameterized Skills and Priors8 May
  6. 2023Developing Driving Strategies Efficiently: A Skill-Based Hierarchical Reinforcement Learning Approach4 Feb
  7. 2022Accelerating Reinforcement Learning for Autonomous Driving using Task-Agnostic and Ego-Centric Motion Skills24 Sep

Other references · 9

  1. 2018The highD Dataset: A Drone Dataset of Naturalistic Vehicle Trajectories on German Highways for Validation of Highly Automated Driving Systemscited by 277
  2. 2021Accelerating Robotic Reinforcement Learning via Parameterized Action Primitivescited by 65
  3. 2018Parametrized Deep Q-Networks Learning: Reinforcement Learning with Discrete-Continuous Hybrid Action Spacecited by 58
  4. 2019DAC: The Double Actor-Critic Architecture for Learning Optionscited by 43
  5. 2023Adaptive and Explainable Deployment of Navigation Skills via Hierarchical Deep Reinforcement Learningcited by 11
  6. 2023Skill-Critic: Refining Learned Skills for Hierarchical Reinforcement Learningcited by 6
  7. 2024Discretionary Lane-Change Decision and Control via Parameterized Soft Actor-Critic for Hybrid Action Spacecited by 6
  8. 2025A Champion-level Vision-based Reinforcement Learning Agent for Competitive Racing in Gran Turismo 7cited by 4
  9. 2025Extensive Exploration in Complex Traffic Scenarios using Hierarchical Reinforcement Learningcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2021Spatially and Seamlessly Hierarchical Reinforcement Learning for State Space and Policy space in Autonomous Driving
  2. 2021A Safe Hierarchical Planning Framework for Complex Driving Scenarios based on Reinforcement Learning
  3. 2020Reinforcement Learning based Control of Imitative Policies for Near-Accident Driving
  4. 2025Learning to Drive Safely with Hybrid Options
  5. 2025Scenario-Based Hierarchical Reinforcement Learning for Automated Driving Decision Making
  6. 2021Hierarchical Program-Triggered Reinforcement Learning Agents For Automated Driving
  7. 2023Action and Trajectory Planning for Urban Autonomous Driving with Hierarchical Reinforcement Learning
  8. 2025A Survey of Reinforcement Learning-Based Motion Planning for Autonomous Driving: Lessons Learned from a Driving Task Perspective