# Efficient Reinforcement Learning for Autonomous Driving with Parameterized Skills and Priors

arXiv: https://arxiv.org/abs/2305.04412 · HTML: https://mlsanity.com/paper/2305.04412/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for driving, one of 9 directions in End-to-end autonomous driving.
- Builds on 34 of its 51 references that are in the map.
- First posted on arXiv 8 May 2023.

## Placed in

- End-to-end autonomous driving / [Reinforcement learning for driving](https://mlsanity.com/driving/reinforcement-learning-for-driving/)

## Builds on (most cited first, 30 of 34)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations](https://mlsanity.com/paper/1709.10087/) (2017)
- [AWAC: Accelerating Online Reinforcement Learning with Offline Datasets](https://mlsanity.com/paper/2006.09359/) (2020)
- [The Option-Critic Architecture](https://mlsanity.com/paper/1609.05140/) (2016)
- [Overcoming Exploration in Reinforcement Learning with Demonstrations](https://mlsanity.com/paper/1709.10089/) (2017)
- [Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards](https://mlsanity.com/paper/1707.08817/) (2017)
- [MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning](https://mlsanity.com/paper/2109.12674/) (2021)
- [Accelerating Reinforcement Learning with Learned Skill Priors](https://mlsanity.com/paper/2010.11944/) (2020)
- [Bellman Eluder Dimension: New Rich Classes of RL Problems, and Sample-Efficient Algorithms](https://mlsanity.com/paper/2102.00815/) (2021)
- [Safety-Enhanced Autonomous Driving Using Interpretable Sensor Fusion Transformer](https://mlsanity.com/paper/2207.14024/) (2022)
- [Imitation Is Not Enough: Robustifying Imitation with Reinforcement Learning for Challenging Driving Scenarios](https://mlsanity.com/paper/2212.11419/) (2022)
- [Interpretable End-to-end Urban Autonomous Driving with Latent Deep Reinforcement Learning](https://mlsanity.com/paper/2001.08726/) (2020)
- [CIRL: Controllable Imitative Reinforcement Learning for Vision-based Self-driving](https://mlsanity.com/paper/1807.03776/) (2018)
- [Neural probabilistic motor primitives for humanoid control](https://mlsanity.com/paper/1811.11711/) (2018)
- [From Motor Control to Team Play in Simulated Humanoid Football](https://mlsanity.com/paper/2105.12196/) (2021)
- [Variable Impedance Control in End-Effector Space: An Action Space for Reinforcement Learning in Contact-Rich Tasks](https://mlsanity.com/paper/1906.08880/) (2019)
- [Accelerating Robotic Reinforcement Learning via Parameterized Action Primitives](https://mlsanity.com/paper/2110.15360/) (2021)
- [Reinforcement Learning with Sparse Rewards using Guidance from Offline Demonstration](https://mlsanity.com/paper/2202.04628/) (2022)
- [Learning compositional models of robot skills for task and motion planning](https://mlsanity.com/paper/2006.06444/) (2020)
- [How would surround vehicles move? A Unified Framework for Maneuver Classification and Motion Prediction](https://mlsanity.com/paper/1801.06523/) (2018)
- [Learning to Simulate Self-Driven Particles System with Coordinated Policy Optimization](https://mlsanity.com/paper/2110.13827/) (2021)
- [Improved Deep Reinforcement Learning with Expert Demonstrations for Urban Autonomous Driving](https://mlsanity.com/paper/2102.09243/) (2021)
- [Social Interactions for Autonomous Driving: A Review and Perspectives](https://mlsanity.com/paper/2208.07541/) (2022)
- [Learning Transferable Motor Skills with Hierarchical Latent Mixture Policies](https://mlsanity.com/paper/2112.05062/) (2021)
- [EPSILON: An Efficient Planning System for Automated Vehicles in Highly Interactive Environments](https://mlsanity.com/paper/2108.07993/) (2021)
- [Keyframing the Future: Keyframe Discovery for Visual Prediction and Planning](https://mlsanity.com/paper/1904.05869/) (2019)
- [Transferable and Adaptable Driving Behavior Prediction](https://mlsanity.com/paper/2202.05140/) (2022)
- [Socially-Compatible Behavior Design of Autonomous Vehicles with Verification on Real Human Data](https://mlsanity.com/paper/2010.14712/) (2020)
- [Learning to Drive using Inverse Reinforcement Learning and Deep Q-Networks](https://mlsanity.com/paper/1612.03653/) (2016)

## Built on by (most cited first, 30 of 36)

- [LMDrive: Closed-Loop End-to-End Driving with Large Language Models](https://mlsanity.com/paper/2312.07488/) (2023)
- [OpenEMMA: Open-Source Multimodal Model for End-to-End Autonomous Driving](https://mlsanity.com/paper/2412.15208/) (2024)
- [AutoTrust: Benchmarking Trustworthiness in Large Vision Language Models for Autonomous Driving](https://mlsanity.com/paper/2412.15206/) (2024)
- [SmartRefine: A Scenario-Adaptive Refinement Framework for Efficient Motion Prediction](https://mlsanity.com/paper/2403.11492/) (2024)
- [CarPlanner: Consistent Auto-regressive Trajectory Planning for Large-scale Reinforcement Learning in Autonomous Driving](https://mlsanity.com/paper/2502.19908/) (2025)
- [IRL-VLA: Training an Vision-Language-Action Policy via Reward World Model](https://mlsanity.com/paper/2508.06571/) (2025)
- [Open-sourced Data Ecosystem in Autonomous Driving: the Present and Future](https://mlsanity.com/paper/2312.03408/) (2023)
- [DrivingGen: A Comprehensive Benchmark for Generative Video World Models in Autonomous Driving](https://mlsanity.com/paper/2601.01528/) (2026)
- [A Review of Reward Functions for Reinforcement Learning in the context of Autonomous Driving](https://mlsanity.com/paper/2405.01440/) (2024)
- [Optimizing Autonomous Driving for Safety: A Human-Centric Approach with LLM-Enhanced RLHF](https://mlsanity.com/paper/2406.04481/) (2024)
- [A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning](https://mlsanity.com/paper/2312.07685/) (2023)
- [Synthesizing Programmatic Reinforcement Learning Policies with Large Language Model Guided Search](https://mlsanity.com/paper/2405.16450/) (2024)
- [PathRL: An End-to-End Path Generation Method for Collision Avoidance via Deep Reinforcement Learning](https://mlsanity.com/paper/2310.13295/) (2023)
- [SmartPretrain: Model-Agnostic and Dataset-Agnostic Representation Learning for Motion Prediction](https://mlsanity.com/paper/2410.08669/) (2024)
- [Open-World Skill Discovery from Unsegmented Demonstrations](https://mlsanity.com/paper/2503.10684/) (2025)
- [Trends in Motion Prediction Toward Deployable and Generalizable Autonomy: A Revisit and Perspectives](https://mlsanity.com/paper/2505.09074/) (2025)
- [ForeSight: Multi-View Streaming Joint Object Detection and Trajectory Forecasting](https://mlsanity.com/paper/2508.07089/) (2025)
- [Imagination-Augmented Hierarchical Reinforcement Learning for Safe and Interactive Autonomous Driving in Urban Environments](https://mlsanity.com/paper/2311.10309/) (2023)
- [DriveMind: A Dual Visual Language Model-based Reinforcement Learning Framework for Autonomous Driving](https://mlsanity.com/paper/2506.00819/) (2025)
- [Large-Scale Mixed-Traffic and Intersection Control using Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2504.04691/) (2025)
- [Integrating Expert Guidance for Efficient Learning of Safe Overtaking in Autonomous Driving Using Deep Reinforcement Learning](https://mlsanity.com/paper/2308.09456/) (2023)
- [Act Better by Timing: A timing-Aware Reinforcement Learning for Autonomous Driving](https://mlsanity.com/paper/2406.13223/) (2024)
- [Safe CoR: A Dual-Expert Approach to Integrating Imitation Learning and Safe Reinforcement Learning Using Constraint Rewards](https://mlsanity.com/paper/2407.02245/) (2024)
- [Scenario-Based Hierarchical Reinforcement Learning for Automated Driving Decision Making](https://mlsanity.com/paper/2506.23023/) (2025)
- [CosmosAlign: Adapting a World Foundation Model for Generative Traffic Video Forecasting](https://mlsanity.com/paper/2608.07693/) (2026)
- [AI Agents and Hard Choices](https://mlsanity.com/paper/2504.15304/) (2025)
- [Interactive Double Deep Q-network: Integrating Human Interventions and Evaluative Predictions in Reinforcement Learning of Autonomous Driving](https://mlsanity.com/paper/2505.01440/) (2025)
- [Learning-based Autonomous Oversteer Control and Collision Avoidance](https://mlsanity.com/paper/2505.15275/) (2025)
- [Multi-Timescale Hierarchical Reinforcement Learning for Unified Behavior and Control of Autonomous Driving](https://mlsanity.com/paper/2506.23771/) (2025)
- [ObjectRL: An Object-Oriented Reinforcement Learning Codebase](https://mlsanity.com/paper/2507.03487/) (2025)

## Abstract (arXiv)

When autonomous vehicles are deployed on public roads, they will encounter countless and diverse driving situations. Many manually designed driving policies are difficult to scale to the real world. Fortunately, reinforcement learning has shown great success in many tasks by automatic trial and error. However, when it comes to autonomous driving in interactive dense traffic, RL agents either fail to learn reasonable performance or necessitate a large amount of data. Our insight is that when humans learn to drive, they will 1) make decisions over the high-level skill space instead of the low-level control space and 2) leverage expert prior knowledge rather than learning from scratch. Inspired by this, we propose ASAP-RL, an efficient reinforcement learning algorithm for autonomous driving that simultaneously leverages motion skills and expert priors. We first parameterized motion skills, which are diverse enough to cover various complex driving scenarios and situations. A skill parameter inverse recovery method is proposed to convert expert demonstrations from control space to skill space. A simple but effective double initialization technique is proposed to leverage expert priors while bypassing the issue of expert suboptimality and early performance degradation. We validate our proposed method on interactive dense-traffic driving tasks given simple and sparse rewards. Experimental results show that our method can lead to higher learning efficiency and better driving performance relative to previous methods that exploit skills and priors differently. Code is open-sourced to facilitate further research.