# Trajectory Planning for Autonomous Vehicles Using Hierarchical Reinforcement Learning

arXiv: https://arxiv.org/abs/2011.04752 · HTML: https://mlsanity.com/paper/2011.04752/

## Key facts

- Cited by 17 papers in the map (data through 7 Oct 2026).
- First posted on arXiv 9 Nov 2020.

## Built on by (most cited first, 17 of 17)

- [Action and Trajectory Planning for Urban Autonomous Driving with Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2306.15968/) (2023)
- [Developing Driving Strategies Efficiently: A Skill-Based Hierarchical Reinforcement Learning Approach](https://mlsanity.com/paper/2302.02179/) (2023)
- [Autonomous Driving using Residual Sensor Fusion and Deep Reinforcement Learning](https://mlsanity.com/paper/2312.16620/) (2023)
- [A Survey of Reinforcement Learning-Based Motion Planning for Autonomous Driving: Lessons Learned from a Driving Task Perspective](https://mlsanity.com/paper/2503.23650/) (2025)
- [Automatic Curriculum Learning for Driving Scenarios: Towards Robust and Efficient Reinforcement Learning](https://mlsanity.com/paper/2505.08264/) (2025)
- [Ego-centric Learning of Communicative World Models for Autonomous Driving](https://mlsanity.com/paper/2506.08149/) (2025)
- [Act Better by Timing: A timing-Aware Reinforcement Learning for Autonomous Driving](https://mlsanity.com/paper/2406.13223/) (2024)
- [Balancing Progress and Safety: A Novel Risk-Aware Objective for RL in Autonomous Driving](https://mlsanity.com/paper/2505.06737/) (2025)
- [GenAI-based Multi-Agent Reinforcement Learning towards Distributed Agent Intelligence: A Generative-RL Agent Perspective](https://mlsanity.com/paper/2507.09495/) (2025)
- [A Review of Learning-Based Motion Planning: Toward a Data-Driven Optimal Control Approach](https://mlsanity.com/paper/2512.11944/) (2025)
- [Predictive Maneuver Planning with Deep Reinforcement Learning (PMP-DRL) for comfortable and safe autonomous driving](https://mlsanity.com/paper/2306.09055/) (2023)
- [Bidirectional Reactive Programming for Machine Learning](https://mlsanity.com/paper/2311.16977/) (2023)
- [Enhancing High-Speed Cruising Performance of Autonomous Vehicles through Integrated Deep Reinforcement Learning Framework](https://mlsanity.com/paper/2404.14713/) (2024)
- [CoMoCAVs: Cohesive Decision-Guided Motion Planning for Connected and Autonomous Vehicles with Multi-Policy Reinforcement Learning](https://mlsanity.com/paper/2507.14903/) (2025)
- [Learning to Drive Safely with Hybrid Options](https://mlsanity.com/paper/2510.24674/) (2025)
- [Planning Autonomous Vehicle Maneuvering in Work Zones Through Game-Theoretic Trajectory Generation](https://mlsanity.com/paper/2603.19556/) (2026)
- [Hierarchical Decision Making with Structured Policies: A Principled Design via Inverse Optimization](https://mlsanity.com/paper/2606.28764/) (2026)

## Abstract (arXiv)

Planning safe trajectories under uncertain and dynamic conditions makes the autonomous driving problem significantly complex. Current sampling-based methods such as Rapidly Exploring Random Trees (RRTs) are not ideal for this problem because of the high computational cost. Supervised learning methods such as Imitation Learning lack generalization and safety guarantees. To address these problems and in order to ensure a robust framework, we propose a Hierarchical Reinforcement Learning (HRL) structure combined with a Proportional-Integral-Derivative (PID) controller for trajectory planning. HRL helps divide the task of autonomous vehicle driving into sub-goals and supports the network to learn policies for both high-level options and low-level trajectory planner choices. The introduction of sub-goals decreases convergence time and enables the policies learned to be reused for other scenarios. In addition, the proposed planner is made robust by guaranteeing smooth trajectories and by handling the noisy perception system of the ego-car. The PID controller is used for tracking the waypoints, which ensures smooth trajectories and reduces jerk. The problem of incomplete observations is handled by using a Long-Short-Term-Memory (LSTM) layer in the network. Results from the high-fidelity CARLA simulator indicate that the proposed method reduces convergence time, generates smoother trajectories, and is able to handle dynamic surroundings and noisy observations.