# Mind the Gap! A Study on the Transferability of Virtual vs Physical-world Testing of Autonomous Driving Systems

arXiv: https://arxiv.org/abs/2112.11255 · HTML: https://mlsanity.com/paper/2112.11255/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 83 references that are in the map.
- First posted on arXiv 21 Dec 2021.

## Builds on (most cited first, 30 of 31)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [End to End Learning for Self-Driving Cars](https://mlsanity.com/paper/1604.07316/) (2016)
- [Deep Reinforcement Learning for Autonomous Driving: A Survey](https://mlsanity.com/paper/2002.00444/) (2020)
- [Unsupervised Image-to-Image Translation Networks](https://mlsanity.com/paper/1703.00848/) (2017)
- [DeepXplore: Automated Whitebox Testing of Deep Learning Systems](https://mlsanity.com/paper/1705.06640/) (2017)
- [DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Cars](https://mlsanity.com/paper/1708.08559/) (2017)
- [Pros and Cons of GAN Evaluation Measures](https://mlsanity.com/paper/1802.03446/) (2018)
- [Taxonomy of Real Faults in Deep Learning Systems](https://mlsanity.com/paper/1910.11015/) (2019)
- [An Analysis of Adversarial Attacks and Defenses on Autonomous Driving Models](https://mlsanity.com/paper/2002.02175/) (2020)
- [SurfelGAN: Synthesizing Realistic Sensor Data for Autonomous Driving](https://mlsanity.com/paper/2005.03844/) (2020)
- [On Offline Evaluation of Vision-based Driving Models](https://mlsanity.com/paper/1809.04843/) (2018)
- [Uncertainty Quantification with Statistical Guarantees in End-to-End Autonomous Driving Control](https://mlsanity.com/paper/1909.09884/) (2019)
- [A Probabilistic Graphical Model Foundation for Enabling Predictive Digital Twins at Scale](https://mlsanity.com/paper/2012.05841/) (2020)
- [Misbehaviour Prediction for Autonomous Driving Systems](https://mlsanity.com/paper/1910.04443/) (2019)
- [DeepBillboard: Systematic Physical-World Testing of Autonomous Driving Systems](https://mlsanity.com/paper/1812.10812/) (2018)
- [Model-based Exploration of the Frontier of Behaviours for Deep Learning System Testing](https://mlsanity.com/paper/2007.02787/) (2020)
- [Comparing Offline and Online Testing of Deep Neural Networks: An Autonomous Car Case Study](https://mlsanity.com/paper/1912.00805/) (2019)
- [Adversarial Driving: Attacking End-to-End Autonomous Driving](https://mlsanity.com/paper/2103.09151/) (2021)
- [MuSHR: A Low-Cost, Open-Source Robotic Racecar for Education and Research](https://mlsanity.com/paper/1908.08031/) (2019)
- [DeepRacer: Educational Autonomous Racing Platform for Experimentation with Sim2Real Reinforcement Learning](https://mlsanity.com/paper/1911.01562/) (2019)
- [Digital Twins Are Not Monozygotic -- Cross-Replicating ADAS Testing in Two Industry-Grade Automotive Simulators](https://mlsanity.com/paper/2012.06822/) (2020)
- [DeepPicar: A Low-cost Deep Neural Network-based Autonomous Car](https://mlsanity.com/paper/1712.08644/) (2017)
- [A Fleet of Miniature Cars for Experiments in Cooperative Driving](https://mlsanity.com/paper/1902.06133/) (2019)
- [Robotics Software Engineering: A Perspective from the Service Robotics Domain](https://mlsanity.com/paper/2006.10608/) (2020)
- [Self-driving scale car trained by Deep reinforcement learning](https://mlsanity.com/paper/1909.03467/) (2019)
- [Learning to Drive (L2D) as a Low-Cost Benchmark for Real-World Reinforcement Learning](https://mlsanity.com/paper/2008.00715/) (2020)
- [Multi-Fidelity Digital Twins: a Means for Better Cyber-Physical Systems Testing?](https://mlsanity.com/paper/2101.05697/) (2021)

## Built on by (most cited first, 30 of 37)

- [Recent Advancements in End-to-End Autonomous Driving using Deep Learning: A Survey](https://mlsanity.com/paper/2307.04370/) (2023)
- [World Models for Autonomous Driving: An Initial Survey](https://mlsanity.com/paper/2403.02622/) (2024)
- [Towards Knowledge-driven Autonomous Driving](https://mlsanity.com/paper/2312.04316/) (2023)
- [Generative AI-empowered Simulation for Autonomous Driving in Vehicular Mixed Reality Metaverses](https://mlsanity.com/paper/2302.08418/) (2023)
- [Ergo, SMIRK is Safe: A Safety Case for a Machine Learning Component in a Pedestrian Automatic Emergency Brake System](https://mlsanity.com/paper/2204.07874/) (2022)
- [Testing of Deep Reinforcement Learning Agents with Surrogate Models](https://mlsanity.com/paper/2305.12751/) (2023)
- [Predicting Safety Misbehaviours in Autonomous Driving Systems using Uncertainty Quantification](https://mlsanity.com/paper/2404.18573/) (2024)
- [Efficient Domain Augmentation for Autonomous Driving Testing Using Diffusion Models](https://mlsanity.com/paper/2409.13661/) (2024)
- [Two is Better Than One: Digital Siblings to Improve Autonomous Driving Testing](https://mlsanity.com/paper/2305.08060/) (2023)
- [Boundary State Generation for Testing and Improvement of Autonomous Driving Systems](https://mlsanity.com/paper/2307.10590/) (2023)
- [EpiTESTER: Testing Autonomous Vehicles with Epigenetic Algorithm and Attention Mechanism](https://mlsanity.com/paper/2312.00207/) (2023)
- [Autonomous Driving Small-Scale Cars: A Survey of Recent Development](https://mlsanity.com/paper/2404.06229/) (2024)
- [Sim2Real Diffusion: Leveraging Foundation Vision Language Models for Adaptive Automated Driving](https://mlsanity.com/paper/2507.00236/) (2025)
- [Bridging the Gap between Real-world and Synthetic Images for Testing Autonomous Driving Systems](https://mlsanity.com/paper/2408.13950/) (2024)
- [Benchmarking Image Perturbations for Testing Automated Driving Assistance Systems](https://mlsanity.com/paper/2501.12269/) (2025)
- [On the Need for a Statistical Foundation in Scenario-Based Testing of Autonomous Vehicles](https://mlsanity.com/paper/2505.02274/) (2025)
- [DeepQTest: Testing Autonomous Driving Systems with Reinforcement Learning and Real-world Weather Data](https://mlsanity.com/paper/2310.05170/) (2023)
- [Generative AI for Testing of Autonomous Driving Systems: A Survey](https://mlsanity.com/paper/2508.19882/) (2025)
- [Combating the effects of speed and delays in end-to-end self-driving](https://mlsanity.com/paper/2312.06670/) (2023)
- [TrueCity: Real and Simulated Urban Data for Cross-Domain 3D Scene Understanding](https://mlsanity.com/paper/2511.07007/) (2025)
- [Behavioural gap assessment of human-vehicle interaction in real and virtual reality-based scenarios in autonomous driving](https://mlsanity.com/paper/2407.04070/) (2024)
- [Behavioral Cloning Models Reality Check for Autonomous Driving](https://mlsanity.com/paper/2409.07218/) (2024)
- [Navigating the Edge with the State-of-the-Art Insights into Corner Case Identification and Generation for Enhanced Autonomous Vehicle Safety](https://mlsanity.com/paper/2503.00077/) (2025)
- [Simulator Ensembles for Trustworthy Autonomous Driving Systems Testing](https://mlsanity.com/paper/2503.08936/) (2025)
- [Offline Reinforcement Learning using Human-Aligned Reward Labeling for Autonomous Emergency Braking in Occluded Pedestrian Crossing](https://mlsanity.com/paper/2504.08704/) (2025)
- [Digital Twin Technologies in Predictive Maintenance: Enabling Transferability via Sim-to-Real and Real-to-Sim Transfer](https://mlsanity.com/paper/2507.18449/) (2025)
- [A Multi-Modality Evaluation of the Reality Gap in Autonomous Driving Systems](https://mlsanity.com/paper/2509.22379/) (2025)
- [Advancing Autonomous Driving System Testing: Demands, Challenges, and Future Directions](https://mlsanity.com/paper/2512.11887/) (2025)
- [Quantifying Fidelity: A Decisive Feature Approach to Comparing Synthetic and Real Imagery](https://mlsanity.com/paper/2512.16468/) (2025)
- [GAN-enhanced Simulation-driven DNN Testing in Absence of Ground Truth](https://mlsanity.com/paper/2503.15953/) (2025)

## Abstract (arXiv)

Safe deployment of self-driving cars (SDC) necessitates thorough simulated and in-field testing. Most testing techniques consider virtualized SDCs within a simulation environment, whereas less effort has been directed towards assessing whether such techniques transfer to and are effective with a physical real-world vehicle. In this paper, we shed light on the problem of generalizing testing results obtained in a driving simulator to a physical platform and provide a characterization and quantification of the sim2real gap affecting SDC testing. In our empirical study, we compare SDC testing when deployed on a physical small-scale vehicle vs its digital twin. Due to the unavailability of driving quality indicators from the physical platform, we use neural rendering to estimate them through visual odometry, hence allowing full comparability with the digital twin. Then, we investigate the transferability of behavior and failure exposure between virtual and real-world environments, targeting both unintended abnormal test data and intended adversarial examples. Our study shows that, despite the usage of a faithful digital twin, there are still critical shortcomings that contribute to the reality gap between the virtual and physical world, threatening existing testing solutions that only consider virtual SDCs. On the positive side, our results present the test configurations for which physical testing can be avoided, either because their outcome does transfer between virtual and physical environments, or because the uncertainty profiles in the simulator can help predict their outcome in the real world.