# Exiting the Simulation: The Road to Robust and Resilient Autonomous Vehicles at Scale

arXiv: https://arxiv.org/abs/2210.10876 · HTML: https://mlsanity.com/paper/2210.10876/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 33 of its 103 references that are in the map.
- First posted on arXiv 19 Oct 2022.

## Builds on (most cited first, 30 of 33)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [Continual Learning with Deep Generative Replay](https://mlsanity.com/paper/1705.08690/) (2017)
- [End to End Learning for Self-Driving Cars](https://mlsanity.com/paper/1604.07316/) (2016)
- [Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images](https://mlsanity.com/paper/1412.1897/) (2014)
- [End-to-end Driving via Conditional Imitation Learning](https://mlsanity.com/paper/1710.02410/) (2017)
- [A Survey of Deep Learning Techniques for Autonomous Driving](https://mlsanity.com/paper/1910.07738/) (2019)
- [ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst](https://mlsanity.com/paper/1812.03079/) (2018)
- [A Survey of Autonomous Driving: Common Practices and Emerging Technologies](https://mlsanity.com/paper/1906.05113/) (2019)
- [A Survey of Motion Planning and Control Techniques for Self-driving Urban Vehicles](https://mlsanity.com/paper/1604.07446/) (2016)
- [CAD2RL: Real Single-Image Flight without a Single Real Image](https://mlsanity.com/paper/1611.04201/) (2016)
- [On a Formal Model of Safe and Scalable Self-driving Cars](https://mlsanity.com/paper/1708.06374/) (2017)
- [A Survey of Deep Learning Applications to Autonomous Vehicle Control](https://mlsanity.com/paper/1912.10773/) (2019)
- [Sim-to-Real Robot Learning from Pixels with Progressive Nets](https://mlsanity.com/paper/1610.04286/) (2016)
- [Driving Policy Transfer via Modularity and Abstraction](https://mlsanity.com/paper/1804.09364/) (2018)
- [Virtual to Real Reinforcement Learning for Autonomous Driving](https://mlsanity.com/paper/1704.03952/) (2017)
- [Scalable End-to-End Autonomous Vehicle Testing via Rare-event Simulation](https://mlsanity.com/paper/1811.00145/) (2018)
- [Accelerated Evaluation of Automated Vehicles Safety in Lane Change Scenarios Based on Importance Sampling Techniques](https://mlsanity.com/paper/1605.04965/) (2016)
- [A-Fast-RCNN: Hard Positive Generation via Adversary for Object Detection](https://mlsanity.com/paper/1704.03414/) (2017)
- [Learning to Drive from Simulation without Real World Labels](https://mlsanity.com/paper/1812.03823/) (2018)
- [Game-Theoretic Modeling of Driver and Vehicle Interactions for Verification and Validation of Autonomous Vehicle Control Systems](https://mlsanity.com/paper/1608.08589/) (2016)
- [Efficient Black-box Assessment of Autonomous Vehicle Safety](https://mlsanity.com/paper/1912.03618/) (2019)
- [Waymo's Safety Methodologies and Safety Readiness Determinations](https://mlsanity.com/paper/2011.00054/) (2020)
- [Yes, we GAN: Applying Adversarial Techniques for Autonomous Driving](https://mlsanity.com/paper/1902.03442/) (2019)
- [Zero-Shot Autonomous Vehicle Policy Transfer: From Simulation to Real-World via Adversarial Learning](https://mlsanity.com/paper/1903.05252/) (2019)

## Abstract (arXiv)

In the past two decades, autonomous driving has been catalyzed into reality by the growing capabilities of machine learning. This paradigm shift possesses significant potential to transform the future of mobility and reshape our society as a whole. With the recent advances in perception, planning, and control capabilities, autonomous driving technologies are being rolled out for public trials, yet we remain far from being able to rigorously ensure the resilient operations of these systems across the long-tailed nature of the driving environment. Given the limitations of real-world testing, autonomous vehicle simulation stands as the critical component in exploring the edge of autonomous driving capabilities, developing the robust behaviors required for successful real-world operation, and enabling the extraction of hidden risks from these complex systems prior to deployment. This paper presents the current state-of-the-art simulation frameworks and methodologies used in the development of autonomous driving systems, with a focus on outlining how simulation is used to build the resiliency required for real-world operation and the methods developed to bridge the gap between simulation and reality. A synthesis of the key challenges surrounding autonomous driving simulation is presented, specifically highlighting the opportunities to further advance the ability to continuously learn in simulation and effectively transfer the learning into the real-world - enabling autonomous vehicles to exit the guardrails of simulation and deliver robust and resilient operations at scale.