mlsanityData to 7 Oct 2026

Boosting Offline Reinforcement Learning for Autonomous Driving with Hierarchical Latent Skills

arXiv 2309.13614 ↗First posted 24 Sep 2023cs.RO cs.AIcited by 13

On the map: End-to-end autonomous driving → Reinforcement learning for driving

also in 1 more
Abstract (arXiv)

Learning-based vehicle planning is receiving increasing attention with the emergence of diverse driving simulators and large-scale driving datasets. While offline reinforcement learning (RL) is well suited for these safety-critical tasks, it still struggles to plan over extended periods. In this work, we present a skill-based framework that enhances offline RL to overcome the long-horizon vehicle planning challenge. Specifically, we design a variational autoencoder (VAE) to learn skills from offline demonstrations. To mitigate posterior collapse of common VAEs, we introduce a two-branch sequence encoder to capture both discrete options and continuous variations of the complex driving skills. The final policy treats learned skills as actions and can be trained by any off-the-shelf offline RL algorithms. This facilitates a shift in focus from per-step actions to temporally extended skills, thereby enabling long-term reasoning into the future. Extensive results on CARLA prove that our model consistently outperforms strong baselines at both training and new scenarios. Additional visualizations and experiments demonstrate the interpretability and transferability of extracted skills.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for driving, one of 9 directions in End-to-end autonomous driving. Builds on 36 of its 52 references that are in the map. First posted on arXiv 24 Sep 2023.

Builds on

36 of 52 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Accelerating Reinforcement Learning for Autonomous Driving using Task-Agnostic and Ego-Centric Motion Skills24 Sep
  2. 2021Offline Reinforcement Learning for Autonomous Driving with Safety and Exploration Enhancement13 Oct

Other references · 34

  1. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  2. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  4. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  5. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,303
  6. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  7. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  8. 2017Hindsight Experience Replaycited by 1,096
  9. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  10. 2020NVAE: A Deep Hierarchical Variational Autoencodercited by 635
  11. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
  12. 2021Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Datasetcited by 532
22 more
  1. 2021NuPlan: A closed-loop ML-based planning benchmark for autonomous vehiclescited by 485
  2. 2018ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worstcited by 391
  3. 2018Understanding disentangling in β-VAEcited by 332
  4. 2019Learning Latent Plans from Playcited by 310
  5. 2017Advances in Variational Inferencecited by 236
  6. 2020Accelerating Reinforcement Learning with Learned Skill Priorscited by 220
  7. 2021End-to-End Urban Driving by Imitating a Reinforcement Learning Coachcited by 191
  8. 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learningcited by 164
  9. 2022Model-Based Imitation Learning for Urban Drivingcited by 149
  10. 2021Goal-Conditioned Reinforcement Learning with Imagined Subgoalscited by 129
  11. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learningcited by 125
  12. 2019IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Datacited by 85
  13. 2022Latent Plans for Task-Agnostic Offline Reinforcement Learningcited by 81
  14. 2022Hierarchical Planning Through Goal-Conditioned Offline Reinforcement Learningcited by 48
  15. 2018How would surround vehicles move? A Unified Framework for Maneuver Classification and Motion Predictioncited by 46
  16. 2022Hierarchical Model-Based Imitation Learning for Planning in Autonomous Drivingcited by 43
  17. 2016On the Quantitative Analysis of Decoder-Based Generative Modelscited by 40
  18. 2022Skill-based Meta-Reinforcement Learningcited by 36
  19. 2023Rethinking Closed-loop Training for Autonomous Drivingcited by 29
  20. 2022InterSim: Interactive Traffic Simulation via Explicit Relation Modelingcited by 27
  21. 2021UMBRELLA: Uncertainty-Aware Model-Based Offline Reinforcement Learning Leveraging Planningcited by 23
  22. 2022Transferable and Adaptable Driving Behavior Predictioncited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 7; 2026: 2.

13 papers.

Sort:

By area:

  1. 2024Asynchronous Large Language Model Enhanced Planner for Autonomous Drivingcited by 38
  2. 2025Graph-Assisted Stitching for Offline Hierarchical Reinforcement Learningcited by 12
  3. 2024Exploring the limits of Hierarchical World Models in Reinforcement Learningcited by 9
  4. 2024TrajDeleter: Enabling Trajectory Forgetting in Offline Reinforcement Learning Agentscited by 7
  5. 2025VLMPlanner: Integrating Visual Language Models with Motion Planningcited by 6
  6. 2025Model-Based Offline Reinforcement Learning with Adversarial Data Augmentationcited by 2
  7. 2025Multi-Agent Reinforcement Learning-based Cooperative Autonomous Driving in Smart Intersectionscited by 2
  8. 2025Sharpening the Spear: Adaptive Expert-Guided Adversarial Attack Against DRL-based Autonomous Driving Policiescited by 2
  9. 2024Augmented Intelligence in Smart Intersections: Local Digital Twins-Assisted Hybrid Autonomous Drivingcited by 1
  10. 2025Offline Reinforcement Learning using Human-Aligned Reward Labeling for Autonomous Emergency Braking in Occluded Pedestrian Crossingcited by 1
  11. 2025Learning to Drive Safely with Hybrid Optionscited by 0
  12. 2026REAP: Reinforcement-Learning End-to-End Autonomous Parking with Gaussian Splatting Simulator for Real2Sim2Real Transfercited by 0
1 more
  1. 2026A Survey on End-to-End Autonomous Driving Training from the Perspectives of Data, Strategy, and Platformcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Learning robust driving policies without online exploration
  2. 2025From Imitation to Optimization: A Comparative Study of Offline Learning for Autonomous Driving
  3. 2024Latent Safety-Constrained Policy Approach for Safe Offline Reinforcement Learning
  4. 2025Pseudo-Expert Regularized Offline RL for End-to-End Autonomous Driving in Photorealistic Closed-Loop Environments
  5. 2025Automatic Curriculum Learning for Driving Scenarios: Towards Robust and Efficient Reinforcement Learning
  6. 2023Guided Online Distillation: Promoting Safe Reinforcement Learning by Offline Demonstration
  7. 2023Efficient Reinforcement Learning for Autonomous Driving with Parameterized Skills and Priors
  8. 2025LearningFlow: Automated Policy Learning Workflow for Urban Driving with Large Language Models