mlsanityData to 7 Oct 2026

Efficient Deep Reinforcement Learning with Imitative Expert Priors for Autonomous Driving

arXiv 2103.10690 ↗First posted 19 Mar 2021cs.ROcited by 46

On the map: End-to-end autonomous driving → Reinforcement learning for driving

Abstract (arXiv)

Deep reinforcement learning (DRL) is a promising way to achieve human-like autonomous driving. However, the low sample efficiency and difficulty of designing reward functions for DRL would hinder its applications in practice. In light of this, this paper proposes a novel framework to incorporate human prior knowledge in DRL, in order to improve the sample efficiency and save the effort of designing sophisticated reward functions. Our framework consists of three ingredients, namely expert demonstration, policy derivation, and reinforcement learning. In the expert demonstration step, a human expert demonstrates their execution of the task, and their behaviors are stored as state-action pairs. In the policy derivation step, the imitative expert policy is derived using behavioral cloning and uncertainty estimation relying on the demonstration data. In the reinforcement learning step, the imitative expert policy is utilized to guide the learning of the DRL agent by regularizing the KL divergence between the DRL agent's policy and the imitative expert policy. To validate the proposed method in autonomous driving applications, two simulated urban driving scenarios (unprotected left turn and roundabout) are designed. The strengths of our proposed method are manifested by the training results as our method can not only achieve the best performance but also significantly improve the sample efficiency in comparison with the baseline algorithms (particularly 60% improvement compared to soft actor-critic). In testing conditions, the agent trained by our method obtains the highest success rate and shows diverse and human-like driving behaviors as demonstrated by the human expert.

Key facts. Cited by 46 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for driving, one of 9 directions in End-to-end autonomous driving. Builds on 27 of its 37 references that are in the map. First posted on arXiv 19 Mar 2021.

Builds on

27 of 37 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Improved Deep Reinforcement Learning with Expert Demonstrations for Urban Autonomous Driving18 Feb

Other references · 26

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  4. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  5. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  6. 2016Generative Adversarial Imitation Learningcited by 1,576
  7. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  8. 2019Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methodscited by 903
  9. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 698
  10. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  11. 2018ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worstcited by 391
  12. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 302
14 more
  1. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learningcited by 213
  2. 2020SMARTS: Scalable Multi-Agent Reinforcement Learning Training School for Autonomous Drivingcited by 121
  3. 2020Interpretable End-to-end Urban Autonomous Driving with Latent Deep Reinforcement Learningcited by 99
  4. 2018CIRL: Controllable Imitative Reinforcement Learning for Vision-based Self-drivingcited by 96
  5. 2020Deep Reinforcement Learning for Intelligent Transportation Systems: A Surveycited by 94
  6. 2019Urban Driving with Conditional Imitation Learningcited by 84
  7. 2019Uncertainty Quantification with Statistical Guarantees in End-to-End Autonomous Driving Controlcited by 60
  8. 2019Multi-Agent Connected Autonomous Driving using Deep Reinforcement Learningcited by 53
  9. 2020Driving Behavior Modeling using Naturalistic Human Driving Data with Inverse Reinforcement Learningcited by 46
  10. 2020Multi-modal Sensor Fusion-Based Deep Neural Network for End-to-end Autonomous Driving with Scene Understandingcited by 42
  11. 2018Reinforced Imitation: Sample Efficient Deep Reinforcement Learning for Map-less Navigation by Leveraging Prior Demonstrationscited by 35
  12. 2019Driving with Style: Inverse Reinforcement Learning in General-Purpose Planning for Automated Drivingcited by 30
  13. 2020Reinforcement Learning based Control of Imitative Policies for Near-Accident Drivingcited by 23
  14. 2019Autonomous Driving using Safe Reinforcement Learning by Incorporating a Regret-based Human Lane-Changing Decision Modelcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 10; 2023: 8; 2024: 7; 2025: 15; 2026: 3.

46 papers.

Sort:

By area:

  1. 2023Motion Planning for Autonomous Driving: The State of the Art and Future Perspectivescited by 112
  2. 2025RAD: Training an End-to-End Driving Policy via Large-Scale 3DGS-based Reinforcement Learningcited by 76
  3. 2022Differentiable Integrated Motion Prediction and Planning with Learnable Cost Function for Autonomous Drivingcited by 71
  4. 2022Conditional Predictive Behavior Planning with Inverse Reinforcement Learning for Human-like Autonomous Drivingcited by 34
  5. 2023Rethinking Imitation-based Planner for Autonomous Drivingcited by 25
  6. 2022Augmenting Reinforcement Learning with Transformer-based Scene Representation Learning for Decision-making of Autonomous Drivingcited by 24
  7. 2025Reinforced Refinement with Self-Aware Expansion for End-to-End Autonomous Drivingcited by 19
  8. 2024Research on Autonomous Driving Decision-making Strategies based Deep Reinforcement Learningcited by 17
  9. 2022Uncovering Instabilities in Variational-Quantum Deep Q-Networkscited by 14
  10. 2023Hundreds Guide Millions: Adaptive Offline Reinforcement Learning with Expert Guidancecited by 13
  11. 2025TeLL-Drive: Enhancing Autonomous Driving with Teacher LLM-Guided Deep Reinforcement Learningcited by 13
  12. 2024Hybrid Reasoning Based on Large Language Models for Autonomous Car Drivingcited by 11
34 more
  1. 2021Modified DDPG car-following model with a real-world human driving experience with CARLA simulatorcited by 9
  2. 2025Enhancing Autonomous Driving Safety with Collision Scenario Integrationcited by 9
  3. 2024Autonomous Driving at Unsignalized Intersections: A Review of Decision-Making Challenges and Reinforcement Learning-Based Solutionscited by 7
  4. 2025SPEQ: Offline Stabilization Phases for Efficient Q-Learning in High Update-To-Data Ratio Reinforcement Learningcited by 7
  5. 2026RAD-2: Scaling Reinforcement Learning in a Generator-Discriminator Frameworkcited by 7
  6. 2023Learning Interaction-aware Motion Prediction Model for Decision-making in Autonomous Drivingcited by 6
  7. 2022Diverse Imitation Learning via Self-Organizing Generative Modelscited by 5
  8. 2022Safe Decision-making for Lane-change of Autonomous Vehicles via Human Demonstration-aided Reinforcement Learningcited by 5
  9. 2023Multi-Agent Reinforcement Learning: Methods, Applications, Visionary Prospects, and Challengescited by 5
  10. 2025LearningFlow: Automated Policy Learning Workflow for Urban Driving with Large Language Modelscited by 4
  11. 2026Fine-tuning is Not Enough: A Parallel Framework for Collaborative Imitation and Reinforcement Learning in End-to-end Autonomous Drivingcited by 4
  12. 2021Generative Adversarial Imitation Learning for End-to-End Autonomous Driving on Urban Environmentscited by 3
  13. 2022Exploring the trade off between human driving imitation and safety for traffic simulationcited by 3
  14. 2021Evaluating the Robustness of Deep Reinforcement Learning for Autonomous Policies in a Multi-agent Urban Driving Environmentcited by 2
  15. 2022Efficient Bayesian Policy Reuse with a Scalable Observation Model in Deep Reinforcement Learningcited by 2
  16. 2022Sampling Efficient Deep Reinforcement Learning through Preference-Guided Stochastic Explorationcited by 2
  17. 2025A Survey of Reinforcement Learning-Based Motion Planning for Autonomous Driving: Lessons Learned from a Driving Task Perspectivecited by 2
  18. 2025Human-Robot Navigation using Event-based Cameras and Reinforcement Learningcited by 2
  19. 2025Sharpening the Spear: Adaptive Expert-Guided Adversarial Attack Against DRL-based Autonomous Driving Policiescited by 2
  20. 2025LLM-Enhanced Multi-Agent Reinforcement Learning with Expert Workflow for Real-Time P2P Energy Tradingcited by 2
  21. 2023Demonstration-guided Deep Reinforcement Learning for Coordinated Ramp Metering and Perimeter Control in Large Scale Networkscited by 1
  22. 2023Efficient Open-world Reinforcement Learning via Knowledge Distillation and Autonomous Rule Discoverycited by 1
  23. 2023Multi-Agent Probabilistic Ensembles with Trajectory Sampling for Connected Autonomous Vehiclescited by 1
  24. 2024Synergistic Reinforcement and Imitation Learning for Vision-driven Autonomous Flight of UAV Along Rivercited by 1
  25. 2024Act Better by Timing: A timing-Aware Reinforcement Learning for Autonomous Drivingcited by 1
  26. 2025A Review of Learning-Based Motion Planning: Toward a Data-Driven Optimal Control Approachcited by 1
  27. 2022An Open Case-based Reasoning Framework for Personalized On-board Driving Assistance in Risk Scenarioscited by 0
  28. 2024Enhancing Reinforcement Learning Through Guided Searchcited by 0
  29. 2024Reinforced Imitative Trajectory Planning for Urban Automated Drivingcited by 0
  30. 2025Towards Developing Socially Compliant Automated Vehicles: Advances, Expert Insights, and A Conceptual Frameworkcited by 0
  31. 2025Exposing the Copycat Problem of Imitation-based Planner: A Novel Closed-Loop Simulator, Causal Benchmark and Joint IL-RL Baselinecited by 0
  32. 2025Bootstrapping Reinforcement Learning with Sub-optimal Policies for Autonomous Drivingcited by 0
  33. 2025Orchestrate, Generate, Reflect: A VLM-Based Multi-Agent Collaboration Framework for Automated Driving Policy Learningcited by 0
  34. 2026MAGNIFIED: RL Fine-tuning of Multimodal Large Language Models for Motion Planningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021A Survey of Deep RL and IL for Autonomous Driving Policy Learning
  2. 2023Review of Deep Reinforcement Learning for Autonomous Driving
  3. 2023Efficient Reinforcement Learning for Autonomous Driving with Parameterized Skills and Priors
  4. 2021Vision-Based Autonomous Car Racing Using Deep Imitative Reinforcement Learning
  5. 2022Learning to Drive Using Sparse Imitation Reinforcement Learning
  6. 2024CIMRL: Combining IMitation and Reinforcement Learning for Safe Autonomous Driving
  7. 2023A Platform-Agnostic Deep Reinforcement Learning Framework for Effective Sim2Real Transfer towards Autonomous Driving
  8. 2021Human-in-the-Loop Deep Reinforcement Learning with Application to Autonomous Driving