# Multi-Agent Reinforcement Learning for Autonomous Driving: A Survey

arXiv: https://arxiv.org/abs/2408.09675 · HTML: https://mlsanity.com/paper/2408.09675/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for driving, one of 9 directions in End-to-end autonomous driving.
- Builds on 158 of its 247 references that are in the map.
- First posted on arXiv 19 Aug 2024.

## Placed in

- End-to-end autonomous driving / [Reinforcement learning for driving](https://mlsanity.com/driving/reinforcement-learning-for-driving/)

## Builds on (most cited first, 30 of 158)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning](https://mlsanity.com/paper/1805.04687/) (2018)
- [A Comprehensive Survey on Transfer Learning](https://mlsanity.com/paper/1911.02685/) (2019)
- [End to End Learning for Self-Driving Cars](https://mlsanity.com/paper/1604.07316/) (2016)
- [Mastering Atari with Discrete World Models](https://mlsanity.com/paper/2010.02193/) (2020)
- [Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning](https://mlsanity.com/paper/2108.10470/) (2021)
- [The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games](https://mlsanity.com/paper/2103.01955/) (2021)
- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation](https://mlsanity.com/paper/2205.13542/) (2022)
- [Recurrent World Models Facilitate Policy Evolution](https://mlsanity.com/paper/1809.01999/) (2018)
- [Counterfactual Multi-Agent Policy Gradients](https://mlsanity.com/paper/1705.08926/) (2017)

## Built on by (most cited first, 30 of 32)

- [A Comprehensive Survey on Multi-Agent Cooperative Decision-Making: Scenarios, Approaches, Challenges and Perspectives](https://mlsanity.com/paper/2503.13415/) (2025)
- [LLM-based Agentic Reasoning Frameworks: A Survey from Methods to Scenarios](https://mlsanity.com/paper/2508.17692/) (2025)
- [Multi-Agent Autonomous Driving Systems with Large Language Models: A Survey of Recent Advances](https://mlsanity.com/paper/2502.16804/) (2025)
- [Language-Driven Policy Distillation for Cooperative Driving in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2410.24152/) (2024)
- [CoLMDriver: LLM-based Negotiation Benefits Cooperative Autonomous Driving](https://mlsanity.com/paper/2503.08683/) (2025)
- [Speaking the Language of Teamwork: LLM-Guided Credit Assignment in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2502.03723/) (2025)
- [OM2P: Offline Multi-Agent Mean-Flow Policy](https://mlsanity.com/paper/2508.06269/) (2025)
- [Deep Meta Coordination Graphs for Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2502.04028/) (2025)
- [QLLM: Do We Really Need a Mixing Network for Credit Assignment in Multi-Agent Reinforcement Learning?](https://mlsanity.com/paper/2504.12961/) (2025)
- [Automated Vehicles Should be Connected with Natural Language](https://mlsanity.com/paper/2507.01059/) (2025)
- [ProxFly: Robust Control for Close Proximity Quadcopter Flight via Residual Reinforcement Learning](https://mlsanity.com/paper/2409.13193/) (2024)
- [GlobeDiff: State Diffusion Process for Partial Observability in Multi-Agent Systems](https://mlsanity.com/paper/2602.15776/) (2026)
- [DriveZero: End-to-End Driving Beyond Human Demonstrations](https://mlsanity.com/paper/2609.06055/) (2026)
- [A Survey of Reinforcement Learning-Based Motion Planning for Autonomous Driving: Lessons Learned from a Driving Task Perspective](https://mlsanity.com/paper/2503.23650/) (2025)
- [Puzzle it Out: Local-to-Global World Model for Offline Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2601.07463/) (2026)
- [Learning Dexterous Bimanual Catch Skills through Adversarial-Cooperative Heterogeneous-Agent Reinforcement Learning](https://mlsanity.com/paper/2502.11437/) (2025)
- [PolicyEvolve: Evolving Programmatic Policies by LLMs for multi-player games via Population-Based Training](https://mlsanity.com/paper/2509.06053/) (2025)
- [From Solo to Symphony: Orchestrating Multi-Agent Collaboration with Single-Agent Demos](https://mlsanity.com/paper/2511.02762/) (2025)
- [Beyond Monotonicity: Revisiting Factorization Principles in Multi-Agent Q-Learning](https://mlsanity.com/paper/2511.09792/) (2025)
- [Markov Potential Game and Multi-Agent Reinforcement Learning for Autonomous Driving](https://mlsanity.com/paper/2603.19188/) (2026)
- [Learning to Drive by Imitating Surrounding Vehicles](https://mlsanity.com/paper/2503.05997/) (2025)
- [Deep Learning Agents Trained For Avoidance Behave Like Hawks And Doves](https://mlsanity.com/paper/2503.11452/) (2025)
- [Dynamic Sight Range Selection in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2505.12811/) (2025)
- [Taxonomy and Trends in Reinforcement Learning for Robotics and Control Systems: A Structured Review](https://mlsanity.com/paper/2510.21758/) (2025)
- [Agentic AI for Self-Driving Laboratories in Soft Matter: Taxonomy, Benchmarks,and Open Challenges](https://mlsanity.com/paper/2601.17920/) (2026)
- [Characterizing MARL for Energy Control: A Multi-KPI Benchmark on the CityLearn Environment](https://mlsanity.com/paper/2602.19223/) (2026)
- [QSIM: Mitigating Overestimation in Multi-Agent Reinforcement Learning via Action Similarity Weighted Q-Learning](https://mlsanity.com/paper/2602.22786/) (2026)
- [COIN: Collaborative Interaction-Aware Multi-Agent Reinforcement Learning for Self-Driving Systems](https://mlsanity.com/paper/2603.24931/) (2026)
- [Multi-Agent Reinforcement Learning for Safe Autonomous Driving Under Pedestrian Behavioral Uncertainty](https://mlsanity.com/paper/2605.20255/) (2026)
- [Safe and Generalizable Hierarchical Multi-Agent RL via Constraint Manifold Control](https://mlsanity.com/paper/2606.24010/) (2026)

## Abstract (arXiv)

Reinforcement Learning (RL) is a potent tool for sequential decision-making and has achieved performance surpassing human capabilities across many challenging real-world tasks. As the extension of RL in the multi-agent system domain, multi-agent RL (MARL) not only need to learn the control policy but also requires consideration regarding interactions with all other agents in the environment, mutual influences among different system components, and the distribution of computational resources. This augments the complexity of algorithmic design and poses higher requirements on computational resources. Simultaneously, simulators are crucial to obtain realistic data, which is the fundamentals of RL. In this paper, we first propose a series of metrics of simulators and summarize the features of existing benchmarks. Second, to ease comprehension, we recall the foundational knowledge and then synthesize the recently advanced studies of MARL-related autonomous driving and intelligent transportation systems. Specifically, we examine their environmental modeling, state representation, perception units, and algorithm design. Conclusively, we discuss open challenges as well as prospects and opportunities. We hope this paper can help the researchers integrate MARL technologies and trigger more insightful ideas toward the intelligent and autonomous driving.