mlsanityData to 7 Oct 2026

Improved Deep Reinforcement Learning with Expert Demonstrations for Urban Autonomous Driving

arXiv 2102.09243 ↗First posted 18 Feb 2021cs.ROcited by 26

On the map: End-to-end autonomous driving → Reinforcement learning for driving

also in 1 more
Abstract (arXiv)

Learning-based approaches, such as reinforcement learning (RL) and imitation learning (IL), have indicated superiority over rule-based approaches in complex urban autonomous driving environments, showing great potential to make intelligent decisions. However, current RL and IL approaches still have their own drawbacks, such as low data efficiency for RL and poor generalization capability for IL. In light of this, this paper proposes a novel learning-based method that combines deep reinforcement learning and imitation learning from expert demonstrations, which is applied to longitudinal vehicle motion control in autonomous driving scenarios. Our proposed method employs the soft actor-critic and modifies the learning process of the policy network to incorporate both the goals of maximizing reward and imitating the expert. Moreover, an adaptive prioritized experience replay is designed to sample experience from both the agent's self-exploration and expert demonstration, in order to improve sample efficiency. The proposed method is validated in a simulated urban roundabout scenario and compared with various prevailing RL and IL baselines. The results manifest that the proposed method has a faster training speed, as well as better performance in navigating safely and time-efficiently.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for driving, one of 9 directions in End-to-end autonomous driving. Builds on 22 of its 36 references that are in the map. First posted on arXiv 18 Feb 2021.

Builds on

22 of 36 references.

References · 22

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  5. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  6. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  7. 2016Generative Adversarial Imitation Learningcited by 1,576
  8. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  9. 2015Prioritized Experience Replaycited by 1,070
  10. 2017End-to-end Driving via Conditional Imitation Learningcited by 479
  11. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 302
  12. 2016A Survey of Motion Planning and Control Techniques for Self-driving Urban Vehiclescited by 251
10 more
  1. 2016End-to-end Learning of Driving Models from Large-scale Video Datasetscited by 147
  2. 2020Survey of Deep Reinforcement Learning for Motion Planning of Autonomous Vehiclescited by 101
  3. 2018CIRL: Controllable Imitative Reinforcement Learning for Vision-based Self-drivingcited by 96
  4. 2019Model-free Deep Reinforcement Learning for Urban Autonomous Drivingcited by 96
  5. 2017Imitating Driver Behavior with Generative Adversarial Networkscited by 92
  6. 2018Deep Imitative Models for Flexible Inference, Planning, and Controlcited by 75
  7. 2020Driving Behavior Modeling using Naturalistic Human Driving Data with Inverse Reinforcement Learningcited by 46
  8. 2020Multi-modal Sensor Fusion-Based Deep Neural Network for End-to-end Autonomous Driving with Scene Understandingcited by 42
  9. 2019Deep Learning-based Vehicle Behaviour Prediction For Autonomous Driving Applications: A Reviewcited by 31
  10. 2017Autonomous Braking System via Deep Reinforcement Learningcited by 18

Built on by

Cited by year

Citing papers per year: 2021: 3; 2022: 8; 2023: 4; 2024: 6; 2025: 5.

26 papers.

Sort:

By area:

  1. 2021Efficient Deep Reinforcement Learning with Imitative Expert Priors for Autonomous Drivingcited by 46
  2. 2023Efficient Reinforcement Learning for Autonomous Driving with Parameterized Skills and Priorscited by 36
  3. 2022Conditional Predictive Behavior Planning with Inverse Reinforcement Learning for Human-like Autonomous Drivingcited by 34
  4. 2023Rethinking Imitation-based Planner for Autonomous Drivingcited by 25
  5. 2022Augmenting Reinforcement Learning with Transformer-based Scene Representation Learning for Decision-making of Autonomous Drivingcited by 24
  6. 2025Reinforced Refinement with Self-Aware Expansion for End-to-End Autonomous Drivingcited by 19
  7. 2021Modified DDPG car-following model with a real-world human driving experience with CARLA simulatorcited by 9
  8. 2025Enhancing Autonomous Driving Safety with Collision Scenario Integrationcited by 9
  9. 2024Efficient and Generalized end-to-end Autonomous Driving System with Latent Deep Reinforcement Learning and Demonstrationscited by 7
  10. 2024Autonomous Driving at Unsignalized Intersections: A Review of Decision-Making Challenges and Reinforcement Learning-Based Solutionscited by 7
  11. 2022Improved Policy Optimization for Online Imitation Learningcited by 6
  12. 2023Learning Interaction-aware Motion Prediction Model for Decision-making in Autonomous Drivingcited by 6
14 more
  1. 2024Pre-trained Transformer-Enabled Strategies with Human-Guided Fine-Tuning for End-to-end Navigation of Autonomous Vehiclescited by 6
  2. 2022Handling Sparse Rewards in Reinforcement Learning Using Model Predictive Controlcited by 4
  3. 2024Beyond Imitation: A Life-long Policy Learning Framework for Path Tracking Control of Autonomous Drivingcited by 4
  4. 2022Exploring the trade off between human driving imitation and safety for traffic simulationcited by 3
  5. 2022Hybrid Deep Reinforcement Learning and Planning for Safe and Comfortable Automated Drivingcited by 3
  6. 2021Evaluating the Robustness of Deep Reinforcement Learning for Autonomous Policies in a Multi-agent Urban Driving Environmentcited by 2
  7. 2024"Give Me an Example Like This": Episodic Active Reinforcement Learning from Demonstrationscited by 2
  8. 2025A Survey of Reinforcement Learning-Based Motion Planning for Autonomous Driving: Lessons Learned from a Driving Task Perspectivecited by 2
  9. 2022A Safety-Critical Decision Making and Control Framework Combining Machine Learning and Rule-based Algorithmscited by 1
  10. 2023Demonstration-guided Deep Reinforcement Learning for Coordinated Ramp Metering and Perimeter Control in Large Scale Networkscited by 1
  11. 2025Restoring Noisy Demonstration for Imitation Learning With Diffusion Modelscited by 1
  12. 2022The Pump Scheduling Problem: A Real-World Scenario for Reinforcement Learningcited by 0
  13. 2024A Framework for testing Federated Learning algorithms using an edge-like environmentcited by 0
  14. 2025Autonomous Vehicle Lateral Control Using Deep Reinforcement Learning with MPC-PID Demonstrationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2020Guided Policy Search Model-based Reinforcement Learning for Urban Autonomous Driving
  2. 2021A Survey of Deep RL and IL for Autonomous Driving Policy Learning
  3. 2020Deep Reinforcement Learning for Autonomous Driving: A Survey
  4. 2023Integration of Reinforcement Learning Based Behavior Planning With Sampling Based Motion Planning for Automated Driving
  5. 2021Vision-Based Autonomous Car Racing Using Deep Imitative Reinforcement Learning
  6. 2022Discrete Control in Real-World Driving Environments using Deep Reinforcement Learning
  7. 2022Learning to Drive Using Sparse Imitation Reinforcement Learning
  8. 2021A Survey of Deep Reinforcement Learning Algorithms for Motion Planning and Control of Autonomous Vehicles