# A Comprehensive Review of Reinforcement Learning for Autonomous Driving in the CARLA Simulator

arXiv: https://arxiv.org/abs/2509.08221 · HTML: https://mlsanity.com/paper/2509.08221/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for driving, one of 9 directions in End-to-end autonomous driving.
- Builds on 37 of its 129 references that are in the map.
- First posted on arXiv 10 Sep 2025.

## Placed in

- End-to-end autonomous driving / [Reinforcement learning for driving](https://mlsanity.com/driving/reinforcement-learning-for-driving/)
- Deep reinforcement learning methods / [RL for autonomous driving and traffic](https://mlsanity.com/deeprl/rl-for-autonomous-driving-and-traffic/)

## Builds on (most cited first, 30 of 37)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Learning Latent Dynamics for Planning from Pixels](https://mlsanity.com/paper/1811.04551/) (2018)
- [When to Trust Your Model: Model-Based Policy Optimization](https://mlsanity.com/paper/1906.08253/) (2019)
- [Reinforcement Learning with Deep Energy-Based Policies](https://mlsanity.com/paper/1702.08165/) (2017)
- [Exploring the Limitations of Behavior Cloning for Autonomous Driving](https://mlsanity.com/paper/1904.08980/) (2019)
- [Deep Reinforcement Learning framework for Autonomous Driving](https://mlsanity.com/paper/1704.02532/) (2017)
- [End-to-End Urban Driving by Imitating a Reinforcement Learning Coach](https://mlsanity.com/paper/2108.08265/) (2021)
- [End-to-End Model-Free Reinforcement Learning for Urban Driving using Implicit Affordances](https://mlsanity.com/paper/1911.10868/) (2019)
- [Learning to drive from a world on rails](https://mlsanity.com/paper/2105.00636/) (2021)
- [Reward (Mis)design for Autonomous Driving](https://mlsanity.com/paper/2104.13906/) (2021)
- [Multi-Agent Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1807.09936/) (2018)
- [MotionNet: Joint Perception and Motion Prediction for Autonomous Driving Based on Bird's Eye View Maps](https://mlsanity.com/paper/2003.06754/) (2020)
- [CIRL: Controllable Imitative Reinforcement Learning for Vision-based Self-driving](https://mlsanity.com/paper/1807.03776/) (2018)
- [Fully Parameterized Quantile Function for Distributional Reinforcement Learning](https://mlsanity.com/paper/1911.02140/) (2019)
- [Conditional Affordance Learning for Driving in Urban Environments](https://mlsanity.com/paper/1806.06498/) (2018)
- [GRI: General Reinforced Imitation and its Application to Vision-Based Autonomous Driving](https://mlsanity.com/paper/2111.08575/) (2021)
- [Iso-Dream: Isolating and Leveraging Noncontrollable Visual Dynamics in World Models](https://mlsanity.com/paper/2205.13817/) (2022)
- [Comprehensive Overview of Reward Engineering and Shaping in Advancing Reinforcement Learning Applications](https://mlsanity.com/paper/2408.10215/) (2024)
- [Domain Adaptation In Reinforcement Learning Via Latent Unified State Representation](https://mlsanity.com/paper/2102.05714/) (2021)
- [A Survey on Simulators for Testing Self-Driving Cars](https://mlsanity.com/paper/2101.05337/) (2021)
- [Vision-Based Autonomous Car Racing Using Deep Imitative Reinforcement Learning](https://mlsanity.com/paper/2107.08325/) (2021)
- [A Review of Reward Functions for Reinforcement Learning in the context of Autonomous Driving](https://mlsanity.com/paper/2405.01440/) (2024)
- [Hierarchical Generative Adversarial Imitation Learning with Mid-level Input Generation for Autonomous Driving on Urban Environments](https://mlsanity.com/paper/2302.04823/) (2023)
- [Hierarchical Program-Triggered Reinforcement Learning Agents For Automated Driving](https://mlsanity.com/paper/2103.13861/) (2021)
- [Integrating Deep Reinforcement Learning with Model-based Path Planners for Automated Driving](https://mlsanity.com/paper/2002.00434/) (2020)
- [RITA: Boost Driving Simulators with Realistic Interactive Traffic Flow](https://mlsanity.com/paper/2211.03408/) (2022)
- [Efficient Learning of Urban Driving Policies Using Bird's-Eye-View State Representations](https://mlsanity.com/paper/2305.19904/) (2023)
- [Navigation In Urban Environments Amongst Pedestrians Using Multi-Objective Deep Reinforcement Learning](https://mlsanity.com/paper/2110.05205/) (2021)

## Built on by (most cited first, 7 of 7)

- [Zero-Human Demonstration End-to-end Autonomous Driving with Trajectory Scorer](https://mlsanity.com/paper/2510.24108/) (2025)
- [More Than Meets the Eye? Uncovering the Reasoning-Planning Disconnect in Training Vision-Language Driving Models](https://mlsanity.com/paper/2510.04532/) (2025)
- [Driving Beyond Privilege: Distilling Dense-Reward Knowledge into Sparse-Reward Policies](https://mlsanity.com/paper/2512.04279/) (2025)
- [Sim2Real-AD: A Modular Sim-to-Real Framework for Deploying VLM-Guided Reinforcement Learning in Real-World Autonomous Driving](https://mlsanity.com/paper/2604.03497/) (2026)
- [Sample Efficient Active Algorithms for Offline Reinforcement Learning](https://mlsanity.com/paper/2602.01260/) (2026)
- [Dreaming Across Towns: Semantic Rollout and Town-Adversarial Regularization for Zero-Shot Held-Out-Town Fixed-Route Driving in CARLA](https://mlsanity.com/paper/2604.27994/) (2026)
- [CORAL: Curriculum-Optimized Reward Adaptation for LiDAR-Based Goal-Directed Urban Driving](https://mlsanity.com/paper/2608.14332/) (2026)

## Abstract (arXiv)

Autonomous-driving research has recently embraced deep Reinforcement Learning (RL) as a promising framework for data-driven decision making, yet a clear picture of how these algorithms are currently employed, benchmarked and evaluated is still missing. This survey fills that gap by systematically analysing around 100 peer-reviewed papers that train, test or validate RL policies inside the open-source CARLA simulator. We first categorize the literature by algorithmic family model-free, model-based, hierarchical, and hybrid and quantify their prevalence, highlighting that more than 80% of existing studies still rely on model-free methods such as DQN, PPO and SAC. Next, we explain the diverse state, action and reward formulations adopted across works, illustrating how choices of sensor modality (RGB, LiDAR, BEV, semantic maps, and carla kinematics states), control abstraction (discrete vs. continuous) and reward shaping are used across various literature. We also consolidate the evaluation landscape by listing the most common metrics (success rate, collision rate, lane deviation, driving score) and the towns, scenarios and traffic configurations used in CARLA benchmarks. Persistent challenges including sparse rewards, sim-to-real transfer, safety guarantees and limited behaviour diversity are distilled into a set of open research questions, and promising directions such as model-based RL, meta-learning and richer multi-agent simulations are outlined. By providing a unified taxonomy, quantitative statistics and a critical discussion of limitations, this review aims to serve both as a reference for newcomers and as a roadmap for advancing RL-based autonomous driving toward real-world deployment.