mlsanityData to 7 Oct 2026

Urban Driver: Learning to Drive from Real-world Demonstrations Using Policy Gradients

arXiv 2109.13333 ↗First posted 27 Sep 2021cs.RO cs.AI cs.CVcited by 122

On the map: End-to-end autonomous driving → Imitation-learning end-to-end planners

Abstract (arXiv)

In this work we are the first to present an offline policy gradient method for learning imitative policies for complex urban driving from a large corpus of real-world demonstrations. This is achieved by building a differentiable data-driven simulator on top of perception outputs and high-fidelity HD maps of the area. It allows us to synthesize new driving experiences from existing demonstrations using mid-level representations. Using this simulator we then train a policy network in closed-loop employing policy gradients. We train our proposed method on 100 hours of expert demonstrations on urban roads and show that it learns complex driving policies that generalize well and can perform a variety of driving maneuvers. We demonstrate this in simulation as well as deploy our model to self-driving vehicles in the real-world. Our method outperforms previously demonstrated state-of-the-art for urban driving scenarios -- all this without the need for complex state perturbations or collecting additional on-policy data during training. We make code and data publicly available.

Key facts. Cited by 122 papers in the map (data through 7 Oct 2026). Placed in Imitation-learning end-to-end planners, one of 9 directions in End-to-end autonomous driving. Builds on 29 of its 47 references that are in the map. First posted on arXiv 27 Sep 2021.

Builds on

29 of 47 references.

References · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2020End-to-End Object Detection with Transformerscited by 7,384
  4. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  5. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  6. 2016Generative Adversarial Imitation Learningcited by 1,576
  7. 2016End to End Learning for Self-Driving Carscited by 1,000
  8. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 698
  9. 2020VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representationcited by 528
  10. 2018ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worstcited by 391
  11. 2020Learning Lane Graph Representations for Motion Forecastingcited by 380
  12. 2016Safe, Multi-Agent, Reinforcement Learning for Autonomous Drivingcited by 301
17 more
  1. 2018Learning to Drive in a Daycited by 266
  2. 2021End-to-end Interpretable Neural Motion Plannercited by 240
  3. 2020One Thousand and One Hours: Self-driving Motion Prediction Datasetcited by 239
  4. 2021MP3: A Unified Model to Map, Perceive, Predict and Plancited by 211
  5. 2021TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviorscited by 192
  6. 2014Recurrent Neural Network Regularizationcited by 189
  7. 2017DART: Noise Injection for Robust Imitation Learningcited by 187
  8. 2015Learning Continuous Control Policies by Stochastic Value Gradientscited by 156
  9. 2018Baidu Apollo EM Motion Plannercited by 99
  10. 2017Deeply AggreVaTeD: Differentiable Imitation Learning for Sequential Predictioncited by 93
  11. 2019Urban Driving with Conditional Imitation Learningcited by 84
  12. 2017Virtual to Real Reinforcement Learning for Autonomous Drivingcited by 79
  13. 2021SimNet: Learning Reactive Self-driving Simulations from Real-world Observationscited by 77
  14. 2019Simulation-based reinforcement learning for real-world autonomous drivingcited by 51
  15. 2019Model-Predictive Policy Learning with Uncertainty Regularization for Driving in Dense Trafficcited by 50
  16. 2020Modeling Human Driving Behavior through Generative Adversarial Imitation Learningcited by 43
  17. 2020The NVIDIA PilotNet Experimentscited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 14; 2023: 17; 2024: 33; 2025: 34; 2026: 24.

122 papers.

Sort:

By area:

  1. 2024NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarkingcited by 343
  2. 2023Parting with Misconceptions about Learning-based Vehicle Motion Planningcited by 238
  3. 2024GenAD: Generative End-to-End Autonomous Drivingcited by 225
  4. 2023OpenOccupancy: A Large Scale Benchmark for Surrounding Semantic Occupancy Perceptioncited by 218
  5. 2025Pseudo-Simulation for Autonomous Drivingcited by 141
  6. 2025Diffusion-Based Planning for Autonomous Driving with Flexible Guidancecited by 138
  7. 2022PlanT: Explainable Planning Transformers via Object-Level Representationscited by 105
  8. 2024Towards learning-based planning:The nuPlan benchmark for real-world autonomous drivingcited by 103
  9. 2022KING: Generating Safety-Critical Driving Scenarios for Robust Imitation via Kinematics Gradientscited by 95
  10. 2025Hydra-MDP++: Advancing End-to-End Driving via Expert-Guided Hydra-Distillationcited by 93
  11. 2024PLUTO: Pushing the Limit of Imitation Learning-based Planning for Autonomous Drivingcited by 84
  12. 2023FusionAD: Multi-modality Fusion for Prediction and Planning Tasks of Autonomous Drivingcited by 67
38 more
  1. 2025Robust Autonomy Emerges from Self-Playcited by 64
  2. 2023TrafficBots: Towards World Models for Autonomous Driving Simulation and Motion Predictioncited by 56
  3. 2024Making Large Language Models Better Planners with Reasoning-Decision Alignmentcited by 51
  4. 2023DTPP: Differentiable Joint Conditional Prediction and Cost Evaluation for Tree Policy Planning in Autonomous Drivingcited by 46
  5. 2023Online Vectorized HD Map Construction using Geometrycited by 42
  6. 2024Diffusion-ES: Gradient-free Planning with Diffusion for Autonomous Driving and Zero-Shot Instruction Followingcited by 41
  7. 2024Solving Motion Planning Tasks with a Scalable Generative Modelcited by 40
  8. 2024Can Vehicle Motion Planning Generalize to Realistic Long-tail Scenarios?cited by 38
  9. 2024PlanAgent: A Multi-modal Large Language Agent for Closed-loop Vehicle Motion Planningcited by 38
  10. 2024Asynchronous Large Language Model Enhanced Planner for Autonomous Drivingcited by 38
  11. 2022Conditional Predictive Behavior Planning with Inverse Reinforcement Learning for Human-like Autonomous Drivingcited by 34
  12. 2025CaRL: Learning Scalable Planning Policies with Simple Rewardscited by 34
  13. 2025Vision-Language-Action Models for Autonomous Driving: Past, Present, and Futurecited by 32
  14. 2025CarPlanner: Consistent Auto-regressive Trajectory Planning for Large-scale Reinforcement Learning in Autonomous Drivingcited by 31
  15. 2023The Integration of Prediction and Planning in Deep Learning Automated Driving Systems: A Reviewcited by 29
  16. 2024Inverse-RLignment: Large Language Model Alignment from Demonstrations through Inverse Reinforcement Learningcited by 27
  17. 2023Rethinking Imitation-based Planner for Autonomous Drivingcited by 25
  18. 2022Are We Ready for Vision-Centric Driving Streaming Perception? The ASAP Benchmarkcited by 24
  19. 2022TrajGen: Generating Realistic and Diverse Trajectories with Reactive and Feasible Agent Behaviors for Autonomous Drivingcited by 23
  20. 2023Large Trajectory Models are Scalable Motion Predictors and Plannerscited by 23
  21. 2025Flow Matching-Based Autonomous Driving Planning with Advanced Interactive Behavior Modelingcited by 22
  22. 2025Plan-R1: Safe and Feasible Trajectory Planning as Language Modelingcited by 19
  23. 2022Driving in Real Life with Inverse Reinforcement Learningcited by 18
  24. 2024Hybrid-Prediction Integrated Planning for Autonomous Drivingcited by 18
  25. 2023CCIL: Context-conditioned imitation learning for urban drivingcited by 12
  26. 2024Supervised Fine-Tuning as Inverse Reinforcement Learningcited by 12
  27. 2024Diffusion Imitation from Observationcited by 12
  28. 2024CALMM-Drive: Confidence-Aware Autonomous Driving with Large Multimodal Modelcited by 12
  29. 2023End-To-End Planning of Autonomous Driving in Industry and Academia: 2022-2023cited by 11
  30. 2024Instruct Large Language Models to Drive like Humanscited by 11
  31. 2024Planning with Adaptive World Models for Autonomous Drivingcited by 11
  32. 2023Editing Driver Character: Socially-Controllable Behavior Generation for Interactive Traffic Simulationcited by 10
  33. 2024LLM4AD: Large Language Models for Autonomous Driving -- Concept, Review, Benchmark, Experiments, and Future Trendscited by 10
  34. 2023Residual Q-Learning: Offline and Online Policy Customization without Valuecited by 9
  35. 2023Learning to Drive Anywherecited by 9
  36. 2024Learning Multiple Probabilistic Decisions from Latent World Model in Autonomous Drivingcited by 8
  37. 2025TrajDiff: End-to-end Autonomous Driving without Perception Annotationcited by 8
  38. 2025LiloDriver: A Lifelong Learning Framework for Closed-loop Motion Planning in Long-tail Autonomous Driving Scenarioscited by 7

50 most cited of 122 All 122 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Learning to Drive from a World Model
  2. 2019Deep Imitation Learning for Autonomous Driving in Generic Urban Scenarios with Enhanced Safety
  3. 2022Model-Based Imitation Learning for Urban Driving
  4. 2025RoaD: Rollouts as Demonstrations for Closed-Loop Supervised Fine-Tuning of Autonomous Driving Policies
  5. 2021Improved Deep Reinforcement Learning with Expert Demonstrations for Urban Autonomous Driving
  6. 2025Learning to Drive by Imitating Surrounding Vehicles
  7. 2022SHAIL: Safety-Aware Hierarchical Adversarial Imitation Learning for Autonomous Driving in Urban Environments
  8. 2021Learning Interactive Driving Policies via Data-driven Simulation