# Building reliable sim driving agents by scaling self-play

arXiv: https://arxiv.org/abs/2502.14706 · HTML: https://mlsanity.com/paper/2502.14706/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 42 references that are in the map.
- First posted on arXiv 20 Feb 2025.

## Builds on (most cited first, 26 of 26)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Dataset](https://mlsanity.com/paper/2104.10133/) (2021)
- [MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning](https://mlsanity.com/paper/2109.12674/) (2021)
- [Wayformer: Motion Forecasting via Simple & Efficient Attention Networks](https://mlsanity.com/paper/2207.05844/) (2022)
- [Waymax: An Accelerated, Data-Driven Simulator for Large-Scale Autonomous Driving Research](https://mlsanity.com/paper/2310.08710/) (2023)
- [Mastering the Game of Stratego with Model-Free Multiagent Reinforcement Learning](https://mlsanity.com/paper/2206.15378/) (2022)
- [The Waymo Open Sim Agents Challenge](https://mlsanity.com/paper/2305.12032/) (2023)
- [BITS: Bi-level Imitation for Traffic Simulation](https://mlsanity.com/paper/2208.12403/) (2022)
- [A Survey of Algorithms for Black-Box Safety Validation of Cyber-Physical Systems](https://mlsanity.com/paper/2005.02979/) (2020)
- [Robust Autonomy Emerges from Self-Play](https://mlsanity.com/paper/2502.03349/) (2025)
- [Trajeglish: Traffic Modeling as Next-Token Prediction](https://mlsanity.com/paper/2312.04535/) (2023)
- [Nocturne: a scalable driving benchmark for bringing multi-agent learning one step closer to the real world](https://mlsanity.com/paper/2206.09889/) (2022)
- [BehaviorGPT: Smart Agent Simulation for Autonomous Driving with Next-Patch Prediction](https://mlsanity.com/paper/2405.17372/) (2024)
- [Mastering the Game of No-Press Diplomacy via Human-Regularized Reinforcement Learning and Planning](https://mlsanity.com/paper/2210.05492/) (2022)
- [GPUDrive: Data-driven, multi-agent driving simulation at 1 million FPS](https://mlsanity.com/paper/2408.01584/) (2024)
- [Closed-Loop Supervised Fine-Tuning of Tokenized Traffic Models](https://mlsanity.com/paper/2412.05334/) (2024)
- [Learning to Simulate Self-Driven Particles System with Coordinated Policy Optimization](https://mlsanity.com/paper/2110.13827/) (2021)
- [Rethinking Closed-loop Training for Autonomous Driving](https://mlsanity.com/paper/2306.15713/) (2023)
- [Learning Realistic Traffic Agents in Closed-loop](https://mlsanity.com/paper/2311.01394/) (2023)
- [Learning to Drive via Asymmetric Self-Play](https://mlsanity.com/paper/2409.18218/) (2024)
- [UniGen: Unified Modeling of Initial Agent States and Trajectories for Generating Autonomous Driving Scenarios](https://mlsanity.com/paper/2405.03807/) (2024)
- [Data Scaling Laws for Imitation Learning-Based End-to-End Autonomous Driving](https://mlsanity.com/paper/2412.02689/) (2024)
- [Modeling road user response timing in naturalistic settings: a surprise-based framework](https://mlsanity.com/paper/2208.08651/) (2022)

## Built on by (most cited first, 21 of 21)

- [SimScale: Learning to Drive via Real-World Simulation at Scale](https://mlsanity.com/paper/2511.23369/) (2025)
- [SPiDR: A Simple Approach for Zero-Shot Safety in Sim-to-Real Transfer](https://mlsanity.com/paper/2509.18648/) (2025)
- [SPACeR: Self-Play Anchoring with Centralized Reference Models](https://mlsanity.com/paper/2510.18060/) (2025)
- [Learning to Drive in New Cities Without Human Demonstrations](https://mlsanity.com/paper/2602.15891/) (2026)
- [Beyond Self-Play and Scale: A Behavior Benchmark for Generalization in Autonomous Driving](https://mlsanity.com/paper/2605.10034/) (2026)
- [Human-like autonomy emerges from self-play and a pinch of human data](https://mlsanity.com/paper/2606.19370/) (2026)
- [Scaling Self-Play for End-to-End Driving](https://mlsanity.com/paper/2606.19641/) (2026)
- [DriveZero: End-to-End Driving Beyond Human Demonstrations](https://mlsanity.com/paper/2609.06055/) (2026)
- [Estimating cognitive biases with attention-aware inverse planning](https://mlsanity.com/paper/2510.25951/) (2025)
- [Post-Training and Test-Time Scaling of Generative Agent Behavior Models for Interactive Autonomous Driving](https://mlsanity.com/paper/2512.13262/) (2025)
- [Heterogeneous Self-Play for Realistic Highway Traffic Simulation](https://mlsanity.com/paper/2604.16406/) (2026)
- [Toward Efficient and Robust Behavior Models for Multi-Agent Driving Simulation](https://mlsanity.com/paper/2512.05812/) (2025)
- [Beyond Self-Play: Hierarchical Reasoning for Continuous Motion in Closed-Loop Traffic Simulation](https://mlsanity.com/paper/2605.09153/) (2026)
- [TerraZero: Procedural Driving Simulation for Zero-Demonstration Self-Play at Scale](https://mlsanity.com/paper/2607.13028/) (2026)
- [Scaling Curriculum Learning For Autonomous Driving](https://mlsanity.com/paper/2608.22549/) (2026)
- [What Emerges and What Breaks in Self-Play Driving](https://mlsanity.com/paper/2608.30819/) (2026)
- [CounterPlay: Counterfactual Post-Training for Self-Play Driving Policies](https://mlsanity.com/paper/2609.21617/) (2026)
- [Oryx: a Scalable Sequence Model for Many-Agent Coordination in Offline MARL](https://mlsanity.com/paper/2505.22151/) (2025)
- [General-purpose LLMs as Models of Human Driver Behavior: The Case of Simplified Merging](https://mlsanity.com/paper/2604.09609/) (2026)
- [What Probing Reveals about Autonomous Driving: Linking Internal Prediction Errors to Ego Planning](https://mlsanity.com/paper/2606.31106/) (2026)
- [Uncertainty-Driven Self-Play for Robust Multi-Agent Driving Policies](https://mlsanity.com/paper/2610.19714/) (2026)

## Abstract (arXiv)

Simulation agents are essential for designing and testing systems that interact with humans, such as autonomous vehicles (AVs). These agents serve various purposes, from benchmarking AV performance to stress-testing system limits, but all applications share one key requirement: reliability. To enable sound experimentation, a simulation agent must behave as intended. It should minimize actions that may lead to undesired outcomes, such as collisions, which can distort the signal-to-noise ratio in analyses. As a foundation for reliable sim agents, we propose scaling self-play to thousands of scenarios on the Waymo Open Motion Dataset under semi-realistic limits on human perception and control. Training from scratch on a single GPU, our agents solve almost the full training set within a day. They generalize to unseen test scenes, achieving a 99.8% goal completion rate with less than 0.8% combined collision and off-road incidents across 10,000 held-out scenarios. Beyond in-distribution generalization, our agents show partial robustness to out-of-distribution scenes and can be fine-tuned in minutes to reach near-perfect performance in such cases. We open-source the pre-trained agents and integrate them with a batched multi-agent simulator. Demonstrations of agent behaviors can be viewed at https://sites.google.com/view/reliable-sim-agents, and we open-source our agents at https://github.com/Emerge-Lab/gpudrive.