# Driving in Dense Traffic with Model-Free Reinforcement Learning arXiv: https://arxiv.org/abs/1909.06710 ยท HTML: https://mlsanity.com/paper/1909.06710/ ## Key facts - Cited by 35 papers in the map (data through 14 Oct 2026). - Placed in World models for autonomous driving, one of 11 directions in World models. - Builds on 2 of its 37 references that are in the map. - First posted on arXiv 15 Sep 2019. ## Placed in - World models / [World models for autonomous driving](https://mlsanity.com/wm/world-models-for-autonomous-driving/) ## Builds on (most cited first, 2 of 2) - [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017) - [Model-free Deep Reinforcement Learning for Urban Autonomous Driving](https://mlsanity.com/paper/1904.09503/) (2019) ## Built on by (most cited first, 30 of 35) - [Understanding World or Predicting Future? A Comprehensive Survey of World Models](https://mlsanity.com/paper/2411.14499/) (2024) - [Motion Planning for Autonomous Driving: The State of the Art and Future Perspectives](https://mlsanity.com/paper/2303.09824/) (2023) - [DRIVE: Deep Reinforced Accident Anticipation with Visual Explanation](https://mlsanity.com/paper/2107.10189/) (2021) - [Challenges and Opportunities in Deep Reinforcement Learning with Graph Neural Networks: A Comprehensive review of Algorithms and Applications](https://mlsanity.com/paper/2206.07922/) (2022) - [PredictionNet: Real-Time Joint Probabilistic Traffic Prediction for Planning, Control, and Simulation](https://mlsanity.com/paper/2109.11094/) (2021) - [Grid-Centric Traffic Scenario Perception for Autonomous Driving: A Comprehensive Review](https://mlsanity.com/paper/2303.01212/) (2023) - [Deep Structured Reactive Planning](https://mlsanity.com/paper/2101.06832/) (2021) - [End-to-End Interactive Prediction and Planning with Optical Flow Distillation for Autonomous Driving](https://mlsanity.com/paper/2104.08862/) (2021) - [DQ-GAT: Towards Safe and Efficient Autonomous Driving with Deep Q-Learning and Graph Attention Networks](https://mlsanity.com/paper/2108.05030/) (2021) - [Accelerating Reinforcement Learning for Autonomous Driving using Task-Agnostic and Ego-Centric Motion Skills](https://mlsanity.com/paper/2209.12072/) (2022) - [Towards Socially Responsive Autonomous Vehicles: A Reinforcement Learning Framework with Driving Priors and Coordination Awareness](https://mlsanity.com/paper/2309.09726/) (2023) - [Carl-Lead: Lidar-based End-to-End Autonomous Driving with Contrastive Deep Reinforcement Learning](https://mlsanity.com/paper/2109.08473/) (2021) - [DDM-Lag : A Diffusion-based Decision-making Model for Autonomous Vehicles with Lagrangian Safety Enhancement](https://mlsanity.com/paper/2401.03629/) (2024) - [Spatiotemporal Costmap Inference for MPC via Deep Inverse Reinforcement Learning](https://mlsanity.com/paper/2201.06539/) (2022) - [Interactive Autonomous Navigation with Internal State Inference and Interactivity Estimation](https://mlsanity.com/paper/2311.16091/) (2023) - [Learning Interaction-aware Guidance Policies for Motion Planning in Dense Traffic Scenarios](https://mlsanity.com/paper/2107.04538/) (2021) - [Predictive Planner for Autonomous Driving with Consistency Models](https://mlsanity.com/paper/2502.08033/) (2025) - [Discretionary Lane-Change Decision and Control via Parameterized Soft Actor-Critic for Hybrid Action Space](https://mlsanity.com/paper/2402.15790/) (2024) - [RoaD: Rollouts as Demonstrations for Closed-Loop Supervised Fine-Tuning of Autonomous Driving Policies](https://mlsanity.com/paper/2512.01993/) (2025) - [Vision Transformer for Learning Driving Policies in Complex Multi-Agent Environments](https://mlsanity.com/paper/2109.06514/) (2021) - [Interaction-Aware Trajectory Planning for Autonomous Vehicles with Analytic Integration of Neural Networks into Model Predictive Control](https://mlsanity.com/paper/2301.05393/) (2023) - [Long and Short-Term Constraints Driven Safe Reinforcement Learning for Autonomous Driving](https://mlsanity.com/paper/2403.18209/) (2024) - [Enhancing PPO with Trajectory-Aware Hybrid Policies](https://mlsanity.com/paper/2502.15968/) (2025) - [HCRMP: A LLM-Hinted Contextual Reinforcement Learning Framework for Autonomous Driving](https://mlsanity.com/paper/2505.15793/) (2025) - [Scaling Self-Play for End-to-End Driving](https://mlsanity.com/paper/2606.19641/) (2026) - [A Hierarchical Hybrid Learning Framework for Multi-agent Trajectory Prediction](https://mlsanity.com/paper/2303.12274/) (2023) - [ADV-0: Closed-Loop Min-Max Adversarial Training for Long-Tail Robustness in Autonomous Driving](https://mlsanity.com/paper/2603.15221/) (2026) - [A Two-stage Based Social Preference Recognition in Multi-Agent Autonomous Driving System](https://mlsanity.com/paper/2310.03303/) (2023) - [Trajectory Planning for Autonomous Vehicle Using Iterative Reward Prediction in Reinforcement Learning](https://mlsanity.com/paper/2404.12079/) (2024) - [A Review of Learning-Based Motion Planning: Toward a Data-Driven Optimal Control Approach](https://mlsanity.com/paper/2512.11944/) (2025) ## Abstract (arXiv) Traditional planning and control methods could fail to find a feasible trajectory for an autonomous vehicle to execute amongst dense traffic on roads. This is because the obstacle-free volume in spacetime is very small in these scenarios for the vehicle to drive through. However, that does not mean the task is infeasible since human drivers are known to be able to drive amongst dense traffic by leveraging the cooperativeness of other drivers to open a gap. The traditional methods fail to take into account the fact that the actions taken by an agent affect the behaviour of other vehicles on the road. In this work, we rely on the ability of deep reinforcement learning to implicitly model such interactions and learn a continuous control policy over the action space of an autonomous vehicle. The application we consider requires our agent to negotiate and open a gap in the road in order to successfully merge or change lanes. Our policy learns to repeatedly probe into the target road lane while trying to find a safe spot to move in to. We compare against two model-predictive control-based algorithms and show that our policy outperforms them in simulation.