mlsanityData to 7 Oct 2026

Model-free Deep Reinforcement Learning for Urban Autonomous Driving

arXiv 1904.09503 ↗First posted 20 Apr 2019cs.LG cs.AI cs.CVcited by 96

On the map: World models → World models for autonomous driving

Abstract (arXiv)

Urban autonomous driving decision making is challenging due to complex road geometry and multi-agent interactions. Current decision making methods are mostly manually designing the driving policy, which might result in sub-optimal solutions and is expensive to develop, generalize and maintain at scale. On the other hand, with reinforcement learning (RL), a policy can be learned and improved automatically without any manual designs. However, current RL methods generally do not work well on complex urban scenarios. In this paper, we propose a framework to enable model-free deep reinforcement learning in challenging urban autonomous driving scenarios. We design a specific input representation and use visual encoding to capture the low-dimensional latent states. Several state-of-the-art model-free deep RL algorithms are implemented into our framework, with several tricks to improve their performance. We evaluate our method in a challenging roundabout task with dense surrounding vehicles in a high-definition driving simulator. The result shows that our method can solve the task well and is significantly better than the baseline.

Key facts. Cited by 96 papers in the map (data through 7 Oct 2026). Placed in World models for autonomous driving, one of 11 directions in World models. Builds on 2 of its 32 references that are in the map. First posted on arXiv 20 Apr 2019.

Builds on

2 of 32 references.

Closest ancestors · 1

Same direction, newest first

  1. 2017CARLA: An Open Urban Driving Simulator10 Nov

Other references · 1

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2019: 1; 2020: 0; 2021: 24; 2022: 16; 2023: 17; 2024: 18; 2025: 16; 2026: 4.

96 papers.

Sort:

By area:

  1. 2021End-to-End Urban Driving by Imitating a Reinforcement Learning Coachcited by 191
  2. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 161
  3. 2024Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methodscited by 114
  4. 2021Reward (Mis)design for Autonomous Drivingcited by 102
  5. 2022SafeBench: A Benchmarking Platform for Safety Evaluation of Autonomous Vehiclescited by 65
  6. 2021A Survey of Deep RL and IL for Autonomous Driving Policy Learningcited by 59
  7. 2024Think2Drive: Efficient Reinforcement Learning by Thinking in Latent World Model for Quasi-Realistic Autonomous Driving (in CARLA-v2)cited by 51
  8. 2022Hierarchical Planning Through Goal-Conditioned Offline Reinforcement Learningcited by 48
  9. 2025Raw2Drive: Reinforcement Learning with Aligned World Models for End-to-End Autonomous Driving (in CARLA v2)cited by 45
  10. 2022Enhance Sample Efficiency and Robustness of End-to-end Urban Autonomous Driving via Semantic Masked World Modelcited by 44
  11. 2019Driving in Dense Traffic with Model-Free Reinforcement Learningcited by 33
  12. 2021Domain Adaptation In Reinforcement Learning Via Latent Unified State Representationcited by 33
38 more
  1. 2024Multi-Agent Reinforcement Learning for Autonomous Driving: A Surveycited by 30
  2. 2021PredictionNet: Real-Time Joint Probabilistic Traffic Prediction for Planning, Control, and Simulationcited by 29
  3. 2021Improved Deep Reinforcement Learning with Expert Demonstrations for Urban Autonomous Drivingcited by 26
  4. 2021Integrated Decision and Control: Towards Interpretable and Computationally Efficient Driving Intelligencecited by 25
  5. 2022Augmenting Reinforcement Learning with Transformer-based Scene Representation Learning for Decision-making of Autonomous Drivingcited by 24
  6. 2021Investigating the Impact of Multi-LiDAR Placement on Object Detection for Autonomous Drivingcited by 22
  7. 2023A Survey on Physics Informed Reinforcement Learning: Review and Open Problemscited by 22
  8. 2025Reinforced Refinement with Self-Aware Expansion for End-to-End Autonomous Drivingcited by 19
  9. 2023Interpretable Reward Redistribution in Reinforcement Learning: A Causal Approachcited by 17
  10. 2024A Review of Reward Functions for Reinforcement Learning in the context of Autonomous Drivingcited by 17
  11. 2022CADRE: A Cascade Deep Reinforcement Learning Framework for Vision-based Autonomous Urban Drivingcited by 16
  12. 2023Safety-aware Causal Representation for Trustworthy Offline Reinforcement Learning in Autonomous Drivingcited by 16
  13. 2024Large Language Models for Human-like Autonomous Driving: A Surveycited by 14
  14. 2021DQ-GAT: Towards Safe and Efficient Autonomous Driving with Deep Q-Learning and Graph Attention Networkscited by 13
  15. 2021Carl-Lead: Lidar-based End-to-End Autonomous Driving with Contrastive Deep Reinforcement Learningcited by 11
  16. 2021A Survey of Deep Reinforcement Learning Algorithms for Motion Planning and Control of Autonomous Vehiclescited by 10
  17. 2023Uncertainty-Aware Decision Transformer for Stochastic Driving Environmentscited by 8
  18. 2023Action and Trajectory Planning for Urban Autonomous Driving with Hierarchical Reinforcement Learningcited by 7
  19. 2024Efficient and Generalized end-to-end Autonomous Driving System with Latent Deep Reinforcement Learning and Demonstrationscited by 7
  20. 2025Model-Based Policy Adaptation for Closed-Loop End-to-End Autonomous Drivingcited by 7
  21. 2023Efficient Learning of Urban Driving Policies Using Bird's-Eye-View State Representationscited by 6
  22. 2025DSADF: Thinking Fast and Slow for Decision Makingcited by 6
  23. 2021Increasing the Efficiency of Policy Learning for Autonomous Vehicles by Multi-Task Representation Learningcited by 5
  24. 2021Encoding Distributional Soft Actor-Critic for Autonomous Driving in Multi-lane Scenarioscited by 5
  25. 2021Vision Transformer for Learning Driving Policies in Complex Multi-Agent Environmentscited by 5
  26. 2021Self-learned Intelligence for Integrated Decision and Control of Automated Vehicles at Signalized Intersectionscited by 5
  27. 2022SHAIL: Safety-Aware Hierarchical Adversarial Imitation Learning for Autonomous Driving in Urban Environmentscited by 5
  28. 2024Computationally Efficient RL under Linear Bellman Completeness for Deterministic Dynamicscited by 5
  29. 2024Hierarchical End-to-End Autonomous Driving: Integrating BEV Perception with Deep Reinforcement Learningcited by 5
  30. 2024VLASCD: A Visual Language Action Model for Simultaneous Chatting and Decision Makingcited by 5
  31. 2021Towards Learning Generalizable Driving Policies from Restricted Latent Representationscited by 4
  32. 2026Scaling Self-Play for End-to-End Drivingcited by 4
  33. 2021Goal-constrained Sparse Reinforcement Learning for End-to-End Drivingcited by 3
  34. 2022Exploring the trade off between human driving imitation and safety for traffic simulationcited by 3
  35. 2022Boosting Exploration in Actor-Critic Algorithms by Incentivizing Plausible Novel Statescited by 3
  36. 2022A Survey on Explainable Reinforcement Learning: Concepts, Algorithms, Challengescited by 3
  37. 2022Hybrid Deep Reinforcement Learning and Planning for Safe and Comfortable Automated Drivingcited by 3
  38. 2023Developing Driving Strategies Efficiently: A Skill-Based Hierarchical Reinforcement Learning Approachcited by 3

50 most cited of 96 All 96 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2021Affordance-based Reinforcement Learning for Urban Driving
  2. 2025A Comprehensive Review of Reinforcement Learning for Autonomous Driving in the CARLA Simulator
  3. 2020Guided Policy Search Model-based Reinforcement Learning for Urban Autonomous Driving
  4. 2020Deep Reinforcement Learning for Autonomous Driving: A Survey
  5. 2023RLAD: Reinforcement Learning from Pixels for Autonomous Driving in Urban Environments
  6. 2024Autonomous Driving at Unsignalized Intersections: A Review of Decision-Making Challenges and Reinforcement Learning-Based Solutions
  7. 2023Review of Deep Reinforcement Learning for Autonomous Driving
  8. 2024Self-Driving Car Racing: Application of Deep Reinforcement Learning