mlsanityData to 14 Oct 2026

Imitation Is Not Enough: Robustifying Imitation with Reinforcement Learning for Challenging Driving Scenarios

arXiv 2212.11419 ↗First posted 21 Dec 2022cs.AI cs.ROcited by 129

On the map: End-to-end autonomous driving → Imitation-learning end-to-end planners

also in 1 more
Abstract (arXiv)

Imitation learning (IL) is a simple and powerful way to use high-quality human driving data, which can be collected at scale, to produce human-like behavior. However, policies based on imitation learning alone often fail to sufficiently account for safety and reliability concerns. In this paper, we show how imitation learning combined with reinforcement learning using simple rewards can substantially improve the safety and reliability of driving policies over those learned from imitation alone. In particular, we train a policy on over 100k miles of urban driving data, and measure its effectiveness in test scenarios grouped by different levels of collision likelihood. Our analysis shows that while imitation can perform well in low-difficulty scenarios that are well-covered by the demonstration data, our proposed approach significantly improves robustness on the most challenging scenarios (over 38% reduction in failures). To our knowledge, this is the first application of a combined imitation and reinforcement learning approach in autonomous driving that utilizes large amounts of real-world human driving data.

Key facts. Cited by 129 papers in the map (data through 14 Oct 2026). Placed in Imitation-learning end-to-end planners, one of 9 directions in End-to-end autonomous driving. Builds on 40 of its 56 references that are in the map. First posted on arXiv 21 Dec 2022.

Builds on

40 of 56 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022Embedding Synthetic Off-Policy Experience for Autonomous Driving via Zero-Shot Curricula2 Dec
  2. 2022Hierarchical Model-Based Imitation Learning for Planning in Autonomous Driving18 Oct
  3. 2021End-to-End Urban Driving by Imitating a Reinforcement Learning Coach18 Aug

Other references · 37

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2017CARLA: An Open Urban Driving Simulatorcited by 2,449
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  4. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  5. 2016Generative Adversarial Imitation Learningcited by 1,597
  6. 2021Perceiver: General Perception with Iterative Attentioncited by 1,104
  7. 2016End to End Learning for Self-Driving Carscited by 1,009
  8. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 899
  9. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  10. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 715
  11. 2021NuPlan: A closed-loop ML-based planning benchmark for autonomous vehiclescited by 493
  12. 2017End-to-end Driving via Conditional Imitation Learningcited by 485
25 more
  1. 2019MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Predictioncited by 420
  2. 2020Learning Lane Graph Representations for Motion Forecastingcited by 382
  3. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 319
  4. 2016Safe, Multi-Agent, Reinforcement Learning for Autonomous Drivingcited by 311
  5. 2021MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learningcited by 290
  6. 2019Causal Confusion in Imitation Learningcited by 288
  7. 2022Wayformer: Motion Forecasting via Simple & Efficient Attention Networkscited by 280
  8. 2018Learning to Drive in a Daycited by 273
  9. 2019INTERACTION Dataset: An INTERnational, Adversarial and Cooperative moTION Dataset in Interactive Driving Scenarios with Semantic Mapscited by 231
  10. 2019PRECOG: PREdiction Conditioned On Goals in Visual Multi-Agent Settingscited by 179
  11. 2021GOHOME: Graph-Oriented Heatmap Output for future Motion Estimationcited by 146
  12. 2021Reward (Mis)design for Autonomous Drivingcited by 104
  13. 2018CIRL: Controllable Imitative Reinforcement Learning for Vision-based Self-drivingcited by 97
  14. 2017Virtual to Real Reinforcement Learning for Autonomous Drivingcited by 80
  15. 2018Deep Imitative Models for Flexible Inference, Planning, and Controlcited by 76
  16. 2021SafetyNet: Safe planning for real-world self-driving vehicles using machine-learned policiescited by 69
  17. 2020Trajectron++: Dynamically-Feasible Trajectory Forecasting With Heterogeneous Datacited by 58
  18. 2022Nocturne: a scalable driving benchmark for bringing multi-agent learning one step closer to the real worldcited by 56
  19. 2018A Reinforcement Learning Based Approach for Automated Lane Change Maneuverscited by 46
  20. 2017Learning to Fly by Crashingcited by 41
  21. 2016Model-based Adversarial Imitation Learningcited by 29
  22. 2021DriverGym: Democratising Reinforcement Learning for Autonomous Drivingcited by 23
  23. 2022Long-Tail Prediction Uncertainty Aware Trajectory Planning for Self-driving Vehiclescited by 22
  24. 2022Critic Sequential Monte Carlocited by 16
  25. 2018Fingerprint Policy Optimisation for Robust Reinforcement Learningcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 15; 2024: 41; 2025: 38; 2026: 35.

129 papers.

Sort:

By area:

  1. 2023Waymax: An Accelerated, Data-Driven Simulator for Large-Scale Autonomous Driving Researchcited by 160
  2. 2025Alpamayo-R1: Bridging Reasoning and Action Prediction for Generalizable Autonomous Driving in the Long Tailcited by 146
  3. 2023The Waymo Open Sim Agents Challengecited by 86
  4. 2024PLUTO: Pushing the Limit of Imitation Learning-based Planning for Autonomous Drivingcited by 84
  5. 2025RAD: Training an End-to-End Driving Policy via Large-Scale 3DGS-based Reinforcement Learningcited by 79
  6. 2023Constrained Decision Transformer for Offline Safe Reinforcement Learningcited by 69
  7. 2025Robust Autonomy Emerges from Self-Playcited by 67
  8. 2023Trajeglish: Traffic Modeling as Next-Token Predictioncited by 60
  9. 2025DiffusionDriveV2: Reinforcement Learning-Constrained Truncated Diffusion Modeling in End-to-End Autonomous Drivingcited by 56
  10. 2023PillarNeXt: Rethinking Network Designs for 3D Object Detection in LiDAR Point Cloudscited by 54
  11. 2024GPUDrive: Data-driven, multi-agent driving simulation at 1 million FPScited by 45
  12. 2024Closed-Loop Supervised Fine-Tuning of Tokenized Traffic Modelscited by 40
38 more
  1. 2025DriveDPO: Policy Learning via Safety DPO For End-to-End Autonomous Drivingcited by 39
  2. 2023Efficient Reinforcement Learning for Autonomous Driving with Parameterized Skills and Priorscited by 36
  3. 2025CaRL: Learning Scalable Planning Policies with Simple Rewardscited by 35
  4. 2025DiWA: Diffusion Policy Adaptation with World Modelscited by 31
  5. 2024CtRL-Sim: Reactive and Controllable Driving Agents with Offline Reinforcement Learningcited by 27
  6. 2024Improving Agent Behaviors with RL Fine-tuning for Autonomous Drivingcited by 27
  7. 2023Rethinking Imitation-based Planner for Autonomous Drivingcited by 25
  8. 2023Learning Realistic Traffic Agents in Closed-loopcited by 24
  9. 2025Discrete Diffusion for Reflective Vision-Language-Action Models in Autonomous Drivingcited by 24
  10. 2023Large Trajectory Models are Scalable Motion Predictors and Plannerscited by 23
  11. 2023When should we prefer Decision Transformers for Offline Reinforcement Learning?cited by 20
  12. 2025Plan-R1: Safe and Feasible Trajectory Planning as Language Modelingcited by 20
  13. 2024LangProp: A code optimization framework using Large Language Models applied to drivingcited by 18
  14. 2024Human-compatible driving partners through data-regularized self-play reinforcement learningcited by 17
  15. 2025DIVER: Reinforced Diffusion Breaks Imitation Bottlenecks in End-to-End Autonomous Drivingcited by 17
  16. 2024Human-centric Reward Optimization for Reinforcement Learning-based Automated Driving using Large Language Modelscited by 15
  17. 2024MOTO: Offline Pre-training to Online Fine-tuning for Model-based Robot Learningcited by 14
  18. 2025DistillDrive: End-to-End Multi-Mode Autonomous Driving Distillation by Isomorphic Hetero-Source Planning Modelcited by 14
  19. 2024TrafficBots V1.5: Traffic Simulation via Conditional VAEs and Transformers with Relative Pose Encodingcited by 13
  20. 2024CIMRL: Combining IMitation and Reinforcement Learning for Safe Autonomous Drivingcited by 12
  21. 2024Learning to Drive via Asymmetric Self-Playcited by 11
  22. 2023Residual Q-Learning: Offline and Online Policy Customization without Valuecited by 10
  23. 2024SEAL: Towards Safe Autonomous Driving via Skill-Enabled Adversary Learning for Closed-Loop Scenario Generationcited by 10
  24. 2025From Screens to Scenes: A Survey of Embodied AI in Healthcarecited by 10
  25. 2025Enhancing Autonomous Driving Safety with Collision Scenario Integrationcited by 9
  26. 2026Beyond Imitation: Reinforcement Learning-Based Sim-Real Co-Training for VLA Modelscited by 9
  27. 2024LASIL: Learner-Aware Supervised Imitation Learning For Long-term Microscopic Traffic Simulationcited by 8
  28. 2026RAD-2: Scaling Reinforcement Learning in a Generator-Discriminator Frameworkcited by 8
  29. 2026World Engine: Towards the Era of Post-Training for Autonomous Drivingcited by 8
  30. 2024Residual-MPPI: Online Policy Customization for Continuous Controlcited by 7
  31. 2024TorchDriveEnv: A Reinforcement Learning Benchmark for Autonomous Driving with Reactive, Realistic, and Diverse Non-Playable Characterscited by 6
  32. 2024Predicting Long-Term Human Behaviors in Discrete Representations via Physics-Guided Diffusioncited by 6
  33. 2024Towards the Transferability of Rewards Recovered via Regularized Inverse Reinforcement Learningcited by 6
  34. 2025Human Implicit Preference-Based Policy Fine-tuning for Multi-Agent Reinforcement Learning in USV Swarmcited by 6
  35. 2025RESample: A Robust Data Augmentation Framework via Exploratory Sampling for Robotic Manipulationcited by 6
  36. 2023Interpretable Motion Planner for Urban Driving via Hierarchical Imitation Learningcited by 4
  37. 2024Validity Learning on Failures: Mitigating the Distribution Shift in Autonomous Vehicle Planningcited by 4
  38. 2024Cross-Modal Self-Supervised Learning with Effective Contrastive Units for LiDAR Point Cloudscited by 4

50 most cited of 129 All 129 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022Exploring the trade off between human driving imitation and safety for traffic simulation
  2. 2022Learning from Demonstrations of Critical Driving Behaviours Using Driver's Risk Field
  3. 2021Safe Imitation Learning on Real-Life Highway Data for Human-like Autonomous Driving
  4. 2022SHAIL: Safety-Aware Hierarchical Adversarial Imitation Learning for Autonomous Driving in Urban Environments
  5. 2025From Imitation to Optimization: A Comparative Study of Offline Learning for Autonomous Driving
  6. 2020Reinforcement Learning based Control of Imitative Policies for Near-Accident Driving
  7. 2021Iterative Imitation Policy Improvement for Interactive Autonomous Driving
  8. 2019Urban Driving with Conditional Imitation Learning