mlsanityData to 7 Oct 2026

Rethinking Closed-loop Training for Autonomous Driving

arXiv 2306.15713 ↗First posted 27 Jun 2023cs.RO cs.CV cs.LGcited by 29

On the map: End-to-end autonomous driving → Imitation-learning end-to-end planners

Abstract (arXiv)

Recent advances in high-fidelity simulators have enabled closed-loop training of autonomous driving agents, potentially solving the distribution shift in training v.s. deployment and allowing training to be scaled both safely and cheaply. However, there is a lack of understanding of how to build effective training benchmarks for closed-loop training. In this work, we present the first empirical study which analyzes the effects of different training benchmark designs on the success of learning agents, such as how to design traffic scenarios and scale training environments. Furthermore, we show that many popular RL algorithms cannot achieve satisfactory performance in the context of autonomous driving, as they lack long-term planning and take an extremely long time to train. To address these issues, we propose trajectory value learning (TRAVL), an RL-based driving agent that performs planning with multistep look-ahead and exploits cheaply generated imagined data for efficient learning. Our experiments show that TRAVL can learn much faster and produce safer maneuvers compared to all the baselines. For more information, visit the project website: https://waabi.ai/research/travl

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Placed in Imitation-learning end-to-end planners, one of 9 directions in End-to-end autonomous driving. Builds on 54 of its 87 references that are in the map. First posted on arXiv 27 Jun 2023.

Builds on

54 of 87 references.

References · 54

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2015Continuous control with deep reinforcement learningcited by 3,279
  6. 2015Trust Region Policy Optimizationcited by 2,503
  7. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  8. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  9. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  10. 2016OpenAI Gymcited by 1,684
  11. 2016Generative Adversarial Imitation Learningcited by 1,576
  12. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
38 more
  1. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  2. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  3. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  4. 2017AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehiclescited by 517
  5. 2021NuPlan: A closed-loop ML-based planning benchmark for autonomous vehiclescited by 485
  6. 2017End-to-end Driving via Conditional Imitation Learningcited by 479
  7. 2018ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worstcited by 391
  8. 2021Multi-Modal Fusion Transformer for End-to-End Autonomous Drivingcited by 372
  9. 2019Learning by Cheatingcited by 361
  10. 2019Exploring the Limitations of Behavior Cloning for Autonomous Drivingcited by 343
  11. 2018An Intriguing Failing of Convolutional Neural Networks and the CoordConv Solutioncited by 302
  12. 2017Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuningcited by 288
  13. 2019Causal Confusion in Imitation Learningcited by 286
  14. 2016Deep Visual Foresight for Planning Robot Motioncited by 276
  15. 2018Learning to Drive in a Daycited by 266
  16. 2021End-to-end Interpretable Neural Motion Plannercited by 240
  17. 2020One Thousand and One Hours: Self-driving Motion Prediction Datasetcited by 239
  18. 2015DeepDriving: Learning Affordance for Direct Perception in Autonomous Drivingcited by 229
  19. 2021TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviorscited by 192
  20. 2018Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansioncited by 167
  21. 2019Benchmarking Model-Based Reinforcement Learningcited by 158
  22. 2018Model-Based Value Estimation for Efficient Model-Free Reinforcement Learningcited by 152
  23. 2020Perceive, Predict, and Plan: Safe Motion Planning Through Interpretable Semantic Representationscited by 150
  24. 2019End-to-End Model-Free Reinforcement Learning for Urban Driving using Implicit Affordancescited by 147
  25. 2020LiDARsim: Realistic LiDAR Simulation by Leveraging the Real Worldcited by 137
  26. 2021AdvSim: Generating Safety-Critical Scenarios for Self-Driving Vehiclescited by 132
  27. 2020SMARTS: Scalable Multi-Agent Reinforcement Learning Training School for Autonomous Drivingcited by 121
  28. 2017Value Prediction Networkcited by 117
  29. 2021Learning to drive from a world on railscited by 114
  30. 2018CIRL: Controllable Imitative Reinforcement Learning for Vision-based Self-drivingcited by 96
  31. 2017Imitating Driver Behavior with Generative Adversarial Networkscited by 92
  32. 2021SimNet: Learning Reactive Self-driving Simulations from Real-world Observationscited by 77
  33. 2018Conditional Affordance Learning for Driving in Urban Environmentscited by 70
  34. 2020DSDNet: Deep Structured self-Driving Networkcited by 61
  35. 2020Multimodal Safety-Critical Scenarios Generation for Decision-Making Algorithms Evaluationcited by 54
  36. 2019Jointly Learnable Behavior and Trajectory Planning for Self-Driving Vehiclescited by 51
  37. 2019Model-Predictive Policy Learning with Uncertainty Regularization for Driving in Dense Trafficcited by 50
  38. 2021Offline Reinforcement Learning for Autonomous Driving with Safety and Exploration Enhancementcited by 28

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 8; 2024: 9; 2025: 8; 2026: 4.

29 papers.

Sort:

By area:

  1. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 485
  2. 2023LanguageMPC: Large Language Models as Decision Makers for Autonomous Drivingcited by 188
  3. 2023Drive Like a Human: Rethinking Autonomous Driving with Large Language Modelscited by 158
  4. 2024AgentsCoDriver: Large Language Model Empowered Collaborative Driving with Lifelong Learningcited by 40
  5. 2025CaRL: Learning Scalable Planning Policies with Simple Rewardscited by 34
  6. 2023The Integration of Prediction and Planning in Deep Learning Automated Driving Systems: A Reviewcited by 29
  7. 2023Towards Knowledge-driven Autonomous Drivingcited by 29
  8. 2023Learning Realistic Traffic Agents in Closed-loopcited by 24
  9. 2024AdvDiffuser: Generating Adversarial Safety-Critical Driving Scenarios via Guided Diffusioncited by 20
  10. 2025Building reliable sim driving agents by scaling self-playcited by 20
  11. 2024Data-Centric Evolution in Autonomous Driving: A Comprehensive Survey of Big Data System, Data Mining, and Closed-Loop Technologiescited by 17
  12. 2023Boosting Offline Reinforcement Learning for Autonomous Driving with Hierarchical Latent Skillscited by 13
17 more
  1. 2024Offboard Occupancy Refinement with Hybrid Propagation for Autonomous Drivingcited by 12
  2. 2024QuAD: Query-based Interpretable Neural Motion Planning for Autonomous Drivingcited by 11
  3. 2024Learning to Drive via Asymmetric Self-Playcited by 11
  4. 2025LD-Scene: LLM-Guided Diffusion for Controllable Generation of Adversarial Safety-Critical Driving Scenarioscited by 9
  5. 2023Uncertainty-Aware Decision Transformer for Stochastic Driving Environmentscited by 8
  6. 2025TANGO: Traversability-Aware Navigation with Local Metric Control for Topological Goalscited by 7
  7. 2024Hierarchical End-to-End Autonomous Driving: Integrating BEV Perception with Deep Reinforcement Learningcited by 5
  8. 2026Scaling Self-Play for End-to-End Drivingcited by 4
  9. 2025Getting SMARTER for Motion Planning in Autonomous Driving Systemscited by 3
  10. 2025Non-differentiable Reward Optimization for Diffusion-based Autonomous Motion Planningcited by 3
  11. 2024Prior-free Balanced Replay: Uncertainty-guided Reservoir Sampling for Long-Tailed Continual Learningcited by 1
  12. 2026ST-Prune: Training-Free Spatio-Temporal Token Pruning for Vision-Language Models in Autonomous Drivingcited by 1
  13. 2024Reinforced Imitative Trajectory Planning for Urban Automated Drivingcited by 0
  14. 2025Active Learning from Scene Embeddings for End-to-End Autonomous Drivingcited by 0
  15. 2025Scalable Offline Metrics for Autonomous Drivingcited by 0
  16. 2026AnchorDrive: LLM Scenario Rollout with Anchor-Guided Diffusion Regeneration for Safety-Critical Scenario Generationcited by 0
  17. 2026A Survey on End-to-End Autonomous Driving Training from the Perspectives of Data, Strategy, and Platformcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Analyzing Closed-loop Training Techniques for Realistic Traffic Agent Models in Autonomous Highway Driving Simulations
  2. 2025Hydra-NeXt: Robust Closed-Loop Driving with Open-Loop Training
  3. 2026Post-Training in End-to-End Autonomous Driving
  4. 2025Automatic Curriculum Learning for Driving Scenarios: Towards Robust and Efficient Reinforcement Learning
  5. 2021Closing the Planning-Learning Loop with Application to Autonomous Driving
  6. 2026Closed-Loop Refinement and Execution for Learned Driving Planners
  7. 2024Enhancing Autonomous Vehicle Training with Language Model Integration and Critical Scenario Generation
  8. 2025RoaD: Rollouts as Demonstrations for Closed-Loop Supervised Fine-Tuning of Autonomous Driving Policies