# Model-free Deep Reinforcement Learning for Urban Autonomous Driving

arXiv: https://arxiv.org/abs/1904.09503 · HTML: https://mlsanity.com/paper/1904.09503/

## Key facts

- Cited by 98 papers in the map (data through 14 Oct 2026).
- Placed in World models for autonomous driving, one of 11 directions in World models.
- Builds on 2 of its 32 references that are in the map.
- First posted on arXiv 20 Apr 2019.

## Placed in

- World models / [World models for autonomous driving](https://mlsanity.com/wm/world-models-for-autonomous-driving/)

## Builds on (most cited first, 2 of 2)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)

## Built on by (most cited first, 30 of 98)

- [End-to-End Urban Driving by Imitating a Reinforcement Learning Coach](https://mlsanity.com/paper/2108.08265/) (2021)
- [Understanding World or Predicting Future? A Comprehensive Survey of World Models](https://mlsanity.com/paper/2411.14499/) (2024)
- [Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methods](https://mlsanity.com/paper/2404.00282/) (2024)
- [Reward (Mis)design for Autonomous Driving](https://mlsanity.com/paper/2104.13906/) (2021)
- [SafeBench: A Benchmarking Platform for Safety Evaluation of Autonomous Vehicles](https://mlsanity.com/paper/2206.09682/) (2022)
- [A Survey of Deep RL and IL for Autonomous Driving Policy Learning](https://mlsanity.com/paper/2101.01993/) (2021)
- [Think2Drive: Efficient Reinforcement Learning by Thinking in Latent World Model for Quasi-Realistic Autonomous Driving (in CARLA-v2)](https://mlsanity.com/paper/2402.16720/) (2024)
- [Hierarchical Planning Through Goal-Conditioned Offline Reinforcement Learning](https://mlsanity.com/paper/2205.11790/) (2022)
- [Raw2Drive: Reinforcement Learning with Aligned World Models for End-to-End Autonomous Driving (in CARLA v2)](https://mlsanity.com/paper/2505.16394/) (2025)
- [Enhance Sample Efficiency and Robustness of End-to-end Urban Autonomous Driving via Semantic Masked World Model](https://mlsanity.com/paper/2210.04017/) (2022)
- [Driving in Dense Traffic with Model-Free Reinforcement Learning](https://mlsanity.com/paper/1909.06710/) (2019)
- [Domain Adaptation In Reinforcement Learning Via Latent Unified State Representation](https://mlsanity.com/paper/2102.05714/) (2021)
- [Multi-Agent Reinforcement Learning for Autonomous Driving: A Survey](https://mlsanity.com/paper/2408.09675/) (2024)
- [PredictionNet: Real-Time Joint Probabilistic Traffic Prediction for Planning, Control, and Simulation](https://mlsanity.com/paper/2109.11094/) (2021)
- [Improved Deep Reinforcement Learning with Expert Demonstrations for Urban Autonomous Driving](https://mlsanity.com/paper/2102.09243/) (2021)
- [Integrated Decision and Control: Towards Interpretable and Computationally Efficient Driving Intelligence](https://mlsanity.com/paper/2103.10290/) (2021)
- [Augmenting Reinforcement Learning with Transformer-based Scene Representation Learning for Decision-making of Autonomous Driving](https://mlsanity.com/paper/2208.12263/) (2022)
- [A Survey on Physics Informed Reinforcement Learning: Review and Open Problems](https://mlsanity.com/paper/2309.01909/) (2023)
- [Investigating the Impact of Multi-LiDAR Placement on Object Detection for Autonomous Driving](https://mlsanity.com/paper/2105.00373/) (2021)
- [Interpretable Reward Redistribution in Reinforcement Learning: A Causal Approach](https://mlsanity.com/paper/2305.18427/) (2023)
- [Reinforced Refinement with Self-Aware Expansion for End-to-End Autonomous Driving](https://mlsanity.com/paper/2506.09800/) (2025)
- [A Review of Reward Functions for Reinforcement Learning in the context of Autonomous Driving](https://mlsanity.com/paper/2405.01440/) (2024)
- [Safety-aware Causal Representation for Trustworthy Offline Reinforcement Learning in Autonomous Driving](https://mlsanity.com/paper/2311.10747/) (2023)
- [CADRE: A Cascade Deep Reinforcement Learning Framework for Vision-based Autonomous Urban Driving](https://mlsanity.com/paper/2202.08557/) (2022)
- [Large Language Models for Human-like Autonomous Driving: A Survey](https://mlsanity.com/paper/2407.19280/) (2024)
- [DQ-GAT: Towards Safe and Efficient Autonomous Driving with Deep Q-Learning and Graph Attention Networks](https://mlsanity.com/paper/2108.05030/) (2021)
- [Carl-Lead: Lidar-based End-to-End Autonomous Driving with Contrastive Deep Reinforcement Learning](https://mlsanity.com/paper/2109.08473/) (2021)
- [A Survey of Deep Reinforcement Learning Algorithms for Motion Planning and Control of Autonomous Vehicles](https://mlsanity.com/paper/2105.14218/) (2021)
- [Uncertainty-Aware Decision Transformer for Stochastic Driving Environments](https://mlsanity.com/paper/2309.16397/) (2023)
- [Action and Trajectory Planning for Urban Autonomous Driving with Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2306.15968/) (2023)

## Abstract (arXiv)

Urban autonomous driving decision making is challenging due to complex road geometry and multi-agent interactions. Current decision making methods are mostly manually designing the driving policy, which might result in sub-optimal solutions and is expensive to develop, generalize and maintain at scale. On the other hand, with reinforcement learning (RL), a policy can be learned and improved automatically without any manual designs. However, current RL methods generally do not work well on complex urban scenarios. In this paper, we propose a framework to enable model-free deep reinforcement learning in challenging urban autonomous driving scenarios. We design a specific input representation and use visual encoding to capture the low-dimensional latent states. Several state-of-the-art model-free deep RL algorithms are implemented into our framework, with several tricks to improve their performance. We evaluate our method in a challenging roundabout task with dense surrounding vehicles in a high-definition driving simulator. The result shows that our method can solve the task well and is significantly better than the baseline.