mlsanityData to 14 Oct 2026

Uncertainty-Driven Self-Play for Robust Multi-Agent Driving Policies

arXiv 2610.19714 ↗First posted 14 Oct 2026cs.RO cs.AI cs.MAcited by 0

On the map: End-to-end autonomous driving → Imitation-learning end-to-end planners also fits: Reinforcement learning for driving

Abstract (arXiv)

Learning robust driving policies that generalize beyond their training domain remains a central challenge in motion planning. We propose an uncertainty-driven self-play scheme, in which half of the agents are rewarded for increasing their neighbors' epistemic uncertainty, while the other half learn to drive safely as the exploring agents continuously seek out their weaknesses. Epistemic uncertainty indicates where the policy is least confident and serves as an automatic proxy for scenario difficulty. Exploring agents use this signal to seek out hard cases, generating a challenging curriculum without manual scenario design. The resulting policy outperforms standard self-play, achieving best-in-class zero-shot transfer to nuplan Val14 (99.12% vs. 97.97%) and Argoverse 2 (82.39% vs. 78.03%) without fine-tuning, while also improving performance over standard self-play on Waymo. Our diversity-driven approach achieves 85% higher LSTM hidden-state coverage than the baseline, indicating a much richer set of behaviors. Further analysis shows that epistemic uncertainty serves as a calibrated difficulty detector, with highest-uncertain agents exhibiting 14.8 times the combined failure rate across collision, off-road, and DNF, of lowest-uncertain agents, enabling automatic prioritization of challenging scenarios without ground-truth labels.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Imitation-learning end-to-end planners, one of 9 directions in End-to-end autonomous driving. Builds on 30 of its 35 references that are in the map. First posted on arXiv 14 Oct 2026.

Builds on

30 of 35 references.

Closest ancestors · 2

Same direction, newest first

  1. 2026CounterPlay: Counterfactual Post-Training for Self-Play Driving Policies18 Sep
  2. 2024CtRL-Sim: Reactive and Controllable Driving Agents with Offline Reinforcement Learning29 Mar

Other references · 28

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  5. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  6. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  7. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,333
  8. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,498
  9. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  10. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
  11. 2023Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecastingcited by 707
  12. 2021Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Datasetcited by 537
16 more
  1. 2021NuPlan: A closed-loop ML-based planning benchmark for autonomous vehiclescited by 493
  2. 2017Robust Adversarial Reinforcement Learningcited by 451
  3. 2018Large-Scale Study of Curiosity-Driven Learningcited by 280
  4. 2020Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Designcited by 275
  5. 2021AdvSim: Generating Safety-Critical Scenarios for Self-Driving Vehiclescited by 137
  6. 2017Uncertainty-Aware Reinforcement Learning for Collision Avoidancecited by 88
  7. 2018Safe Reinforcement Learning with Model Uncertainty Estimatescited by 72
  8. 2025Robust Autonomy Emerges from Self-Playcited by 67
  9. 2024PufferLib: Making Reinforcement Learning Libraries and Environments Play Nicecited by 24
  10. 2025Building reliable sim driving agents by scaling self-playcited by 21
  11. 2023Learning Hierarchical Interactive Multi-Object Search for Mobile Manipulationcited by 14
  12. 2025GraphPilot: Grounded Scene Graph Conditioning for Language-Based Autonomous Drivingcited by 7
  13. 2025Evidential Uncertainty Estimation for Multi-Modal Trajectory Predictioncited by 5
  14. 2025Safety-Critical Traffic Simulation with Adversarial Transfer of Driving Intentionscited by 4
  15. 2026SaFeR: Safety-Critical Scenario Generation for Autonomous Driving Test via Feasibility-Constrained Token Resamplingcited by 2
  16. 2026CoPark: Learning Reactive Parking via Self-Playcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Uncertainty-Aware and Temporally Regulated Expert Advice in Reinforcement Learning for Autonomous Driving
  2. 2023Uncertainty-Aware Decision Transformer for Stochastic Driving Environments
  3. 2024RACER: Epistemic Risk-Sensitive RL Enables Fast Driving with Fewer Crashes
  4. 2026World Models as Adversaries: Multi-Agent Self-Play Fine-Tuning for Robust Motion Planning
  5. 2026TerraZero: Procedural Driving Simulation for Zero-Demonstration Self-Play at Scale
  6. 2022Embedding Synthetic Off-Policy Experience for Autonomous Driving via Zero-Shot Curricula
  7. 2025InDRiVE: Reward-Free World-Model Pretraining for Autonomous Driving via Latent Disagreement
  8. 2025Automatic Curriculum Learning for Driving Scenarios: Towards Robust and Efficient Reinforcement Learning