# Robust Autonomy Emerges from Self-Play

arXiv: https://arxiv.org/abs/2502.03349 · HTML: https://mlsanity.com/paper/2502.03349/

## Key facts

- Cited by 67 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 73 references that are in the map.
- First posted on arXiv 5 Feb 2025.

## Builds on (most cited first, 30 of 45)

- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Deep Sets](https://mlsanity.com/paper/1703.06114/) (2017)
- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [GAIA-1: A Generative World Model for Autonomous Driving](https://mlsanity.com/paper/2309.17080/) (2023)
- [Learning Quadrupedal Locomotion over Challenging Terrain](https://mlsanity.com/paper/2010.11251/) (2020)
- [Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Dataset](https://mlsanity.com/paper/2104.10133/) (2021)
- [Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning](https://mlsanity.com/paper/2109.11978/) (2021)
- [NuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles](https://mlsanity.com/paper/2106.11810/) (2021)
- [TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Driving](https://mlsanity.com/paper/2205.15997/) (2022)
- [Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulation](https://mlsanity.com/paper/2106.13281/) (2021)
- [Population Based Training of Neural Networks](https://mlsanity.com/paper/1711.09846/) (2017)
- [DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Frames](https://mlsanity.com/paper/1911.00357/) (2019)
- [Neural Scene Graphs for Dynamic Scenes](https://mlsanity.com/paper/2011.10379/) (2020)
- [UniSim: A Neural Closed-Loop Sensor Simulator](https://mlsanity.com/paper/2308.01898/) (2023)
- [MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning](https://mlsanity.com/paper/2109.12674/) (2021)
- [Motion Transformer with Global Intention Localization and Local Movement Refinement](https://mlsanity.com/paper/2209.13508/) (2022)
- [Parting with Misconceptions about Learning-based Vehicle Motion Planning](https://mlsanity.com/paper/2306.07962/) (2023)
- [MP3: A Unified Model to Map, Perceive, Predict and Plan](https://mlsanity.com/paper/2101.06806/) (2021)
- [TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviors](https://mlsanity.com/paper/2101.06557/) (2021)
- [Guided Conditional Diffusion for Controllable Traffic Simulation](https://mlsanity.com/paper/2210.17366/) (2022)
- [Waymax: An Accelerated, Data-Driven Simulator for Large-Scale Autonomous Driving Research](https://mlsanity.com/paper/2310.08710/) (2023)
- [Learning from All Vehicles](https://mlsanity.com/paper/2203.11934/) (2022)
- [Mastering the Game of Stratego with Model-Free Multiagent Reinforcement Learning](https://mlsanity.com/paper/2206.15378/) (2022)
- [Imitation Is Not Enough: Robustifying Imitation with Reinforcement Learning for Challenging Driving Scenarios](https://mlsanity.com/paper/2212.11419/) (2022)
- [Urban Driver: Learning to Drive from Real-world Demonstrations Using Policy Gradients](https://mlsanity.com/paper/2109.13333/) (2021)
- [PlanT: Explainable Planning Transformers via Object-Level Representations](https://mlsanity.com/paper/2210.14222/) (2022)
- [Hidden Biases of End-to-End Driving Models](https://mlsanity.com/paper/2306.07957/) (2023)
- [Driving Policy Transfer via Modularity and Abstraction](https://mlsanity.com/paper/1804.09364/) (2018)
- [The Waymo Open Sim Agents Challenge](https://mlsanity.com/paper/2305.12032/) (2023)

## Built on by (most cited first, 30 of 67)

- [Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning](https://mlsanity.com/paper/2504.13818/) (2025)
- [ReSim: Reliable World Simulation for Autonomous Driving](https://mlsanity.com/paper/2506.09981/) (2025)
- [A Survey of Sim-to-Real Methods in RL: Progress, Prospects and Challenges with Foundation Models](https://mlsanity.com/paper/2502.13187/) (2025)
- [SimScale: Learning to Drive via Real-World Simulation at Scale](https://mlsanity.com/paper/2511.23369/) (2025)
- [CaRL: Learning Scalable Planning Policies with Simple Rewards](https://mlsanity.com/paper/2504.17838/) (2025)
- [Discrete Diffusion for Reflective Vision-Language-Action Models in Autonomous Driving](https://mlsanity.com/paper/2509.20109/) (2025)
- [LEAD: Minimizing Learner-Expert Asymmetry in End-to-End Driving](https://mlsanity.com/paper/2512.20563/) (2025)
- [Flow Matching-Based Autonomous Driving Planning with Advanced Interactive Behavior Modeling](https://mlsanity.com/paper/2510.11083/) (2025)
- [Building reliable sim driving agents by scaling self-play](https://mlsanity.com/paper/2502.14706/) (2025)
- [Plan-R1: Safe and Feasible Trajectory Planning as Language Modeling](https://mlsanity.com/paper/2505.17659/) (2025)
- [Reinforced Refinement with Self-Aware Expansion for End-to-End Autonomous Driving](https://mlsanity.com/paper/2506.09800/) (2025)
- [Fail2Drive: Benchmarking Closed-Loop Driving Generalization](https://mlsanity.com/paper/2604.08535/) (2026)
- [Zero-Human Demonstration End-to-end Autonomous Driving with Trajectory Scorer](https://mlsanity.com/paper/2510.24108/) (2025)
- [SPiDR: A Simple Approach for Zero-Shot Safety in Sim-to-Real Transfer](https://mlsanity.com/paper/2509.18648/) (2025)
- [Scalable Decision Making for Games of Imperfect Information](https://mlsanity.com/paper/2511.07312/) (2025)
- [SPACeR: Self-Play Anchoring with Centralized Reference Models](https://mlsanity.com/paper/2510.18060/) (2025)
- [Digital Red Queen: Adversarial Program Evolution in Core War with LLMs](https://mlsanity.com/paper/2601.03335/) (2026)
- [Learning to Drive in New Cities Without Human Demonstrations](https://mlsanity.com/paper/2602.15891/) (2026)
- [Foundation Model Self-Play: Open-Ended Strategy Innovation via Foundation Models](https://mlsanity.com/paper/2507.06466/) (2025)
- [Beyond Self-Play and Scale: A Behavior Benchmark for Generalization in Autonomous Driving](https://mlsanity.com/paper/2605.10034/) (2026)
- [Learning Through Retrospection: Improving Trajectory Prediction for Automated Driving with Error Feedback](https://mlsanity.com/paper/2504.13785/) (2025)
- [RIFT: Group-Relative RL Fine-Tuning for Realistic and Controllable Traffic Simulation](https://mlsanity.com/paper/2505.03344/) (2025)
- [Decentralized Aerial Manipulation of a Cable-Suspended Load using Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2508.01522/) (2025)
- [Do LLM Modules Generalize? A Study on Motion Generation for Autonomous Driving](https://mlsanity.com/paper/2509.02754/) (2025)
- [Agile Flight Emerges from Multi-Agent Competitive Racing](https://mlsanity.com/paper/2512.11781/) (2025)
- [Human-like autonomy emerges from self-play and a pinch of human data](https://mlsanity.com/paper/2606.19370/) (2026)
- [Scaling Self-Play for End-to-End Driving](https://mlsanity.com/paper/2606.19641/) (2026)
- [V-Max: A Reinforcement Learning Framework for Autonomous Driving](https://mlsanity.com/paper/2503.08388/) (2025)
- [Good Actions Succeed, Bad Actions Generalize: A Case Study on Why RL Generalizes Better](https://mlsanity.com/paper/2503.15693/) (2025)
- [123D: Unifying Multi-Modal Autonomous Driving Data at Scale](https://mlsanity.com/paper/2605.08084/) (2026)

## Abstract (arXiv)

Self-play has powered breakthroughs in two-player and multi-player games. Here we show that self-play is a surprisingly effective strategy in another domain. We show that robust and naturalistic driving emerges entirely from self-play in simulation at unprecedented scale -- 1.6~billion~km of driving. This is enabled by Gigaflow, a batched simulator that can synthesize and train on 42 years of subjective driving experience per hour on a single 8-GPU node. The resulting policy achieves state-of-the-art performance on three independent autonomous driving benchmarks. The policy outperforms the prior state of the art when tested on recorded real-world scenarios, amidst human drivers, without ever seeing human data during training. The policy is realistic when assessed against human references and achieves unprecedented robustness, averaging 17.5 years of continuous driving between incidents in simulation.