mlsanityData to 7 Oct 2026

SoftCTRL: Soft conservative KL-control of Transformer Reinforcement Learning for Autonomous Driving

arXiv 2410.22752 ↗First posted 30 Oct 2024cs.RO cs.AIcited by 0

On the map: End-to-end autonomous driving → Reinforcement learning for driving

Abstract (arXiv)

In recent years, motion planning for urban self-driving cars (SDV) has become a popular problem due to its complex interaction of road components. To tackle this, many methods have relied on large-scale, human-sampled data processed through Imitation learning (IL). Although effective, IL alone cannot adequately handle safety and reliability concerns. Combining IL with Reinforcement learning (RL) by adding KL divergence between RL and IL policy to the RL loss can alleviate IL's weakness but suffer from over-conservation caused by covariate shift of IL. To address this limitation, we introduce a method that combines IL with RL using an implicit entropy-KL control that offers a simple way to reduce the over-conservation characteristic. In particular, we validate different challenging simulated urban scenarios from the unseen dataset, indicating that although IL can perform well in imitation tasks, our proposed method significantly improves robustness (over 17% reduction in failures) and generates human-like driving behavior.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning for driving, one of 9 directions in End-to-end autonomous driving. Builds on 41 of its 53 references that are in the map. First posted on arXiv 30 Oct 2024.

Builds on

41 of 53 references.

References · 41

  1. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  2. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  3. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  4. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  5. 2016OpenAI Gymcited by 1,684
  6. 2016Generative Adversarial Imitation Learningcited by 1,576
  7. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  8. 2022Planning-oriented Autonomous Drivingcited by 1,068
  9. 2016End to End Learning for Self-Driving Carscited by 1,000
  10. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  11. 2019Argoverse: 3D Tracking and Forecasting with Rich Mapscited by 728
  12. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
29 more
  1. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  2. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  3. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  4. 2020VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representationcited by 528
  5. 2022TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Drivingcited by 477
  6. 2019MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Predictioncited by 417
  7. 2018ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worstcited by 391
  8. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 322
  9. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 302
  10. 2018Multimodal Trajectory Predictions for Autonomous Driving using Deep Convolutional Networkscited by 289
  11. 2019Causal Confusion in Imitation Learningcited by 286
  12. 2022Wayformer: Motion Forecasting via Simple & Efficient Attention Networkscited by 279
  13. 2022Motion Transformer with Global Intention Localization and Local Movement Refinementcited by 267
  14. 2018Learning to Drive in a Daycited by 266
  15. 2020One Thousand and One Hours: Self-driving Motion Prediction Datasetcited by 239
  16. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learningcited by 213
  17. 2021TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviorscited by 192
  18. 2021End-to-End Urban Driving by Imitating a Reinforcement Learning Coachcited by 191
  19. 2023MTR++: Multi-Agent Motion Prediction with Symmetric Scene Modeling and Guided Intention Queryingcited by 139
  20. 2022Imitation Is Not Enough: Robustifying Imitation with Reinforcement Learning for Challenging Driving Scenarioscited by 124
  21. 2021Urban Driver: Learning to Drive from Real-world Demonstrations Using Policy Gradientscited by 122
  22. 2022PlanT: Explainable Planning Transformers via Object-Level Representationscited by 105
  23. 2021SimNet: Learning Reactive Self-driving Simulations from Real-world Observationscited by 77
  24. 2021Multi-modal Motion Prediction with Transformer-based Neural Network for Autonomous Drivingcited by 73
  25. 2020Munchausen Reinforcement Learningcited by 68
  26. 2019Model-Predictive Policy Learning with Uncertainty Regularization for Driving in Dense Trafficcited by 50
  27. 2018A Reinforcement Learning Based Approach for Automated Lane Change Maneuverscited by 44
  28. 2020Modeling Human Driving Behavior through Generative Adversarial Imitation Learningcited by 43
  29. 2016Model-based Adversarial Imitation Learningcited by 29

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2021Improved Deep Reinforcement Learning with Expert Demonstrations for Urban Autonomous Driving
  2. 2024CIMRL: Combining IMitation and Reinforcement Learning for Safe Autonomous Driving
  3. 2020Guided Policy Search Model-based Reinforcement Learning for Urban Autonomous Driving
  4. 2022Learning to Drive Using Sparse Imitation Reinforcement Learning
  5. 2020Efficient Sampling-Based Maximum Entropy Inverse Reinforcement Learning with Application to Autonomous Driving
  6. 2025From Imitation to Optimization: A Comparative Study of Offline Learning for Autonomous Driving
  7. 2021A Survey of Deep Reinforcement Learning Algorithms for Motion Planning and Control of Autonomous Vehicles
  8. 2020Deep Reinforcement Learning for Autonomous Driving: A Survey