# A Multi-Modality Evaluation of the Reality Gap in Autonomous Driving Systems arXiv: https://arxiv.org/abs/2509.22379 ยท HTML: https://mlsanity.com/paper/2509.22379/ ## Key facts - Cited by 1 papers in the map (data through 14 Oct 2026). - Placed in Adversarial scenarios and safety testing, one of 9 directions in End-to-end autonomous driving. - Builds on 51 of its 84 references that are in the map. - First posted on arXiv 26 Sep 2025. ## Placed in - End-to-end autonomous driving / [Adversarial scenarios and safety testing](https://mlsanity.com/driving/adversarial-scenarios-and-safety-testing/) ## Builds on (most cited first, 30 of 51) - [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014) - [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014) - [End to End Learning for Self-Driving Cars](https://mlsanity.com/paper/1604.07316/) (2016) - [End-to-end Autonomous Driving: Challenges and Frontiers](https://mlsanity.com/paper/2306.16927/) (2023) - [TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Driving](https://mlsanity.com/paper/2205.15997/) (2022) - [A Survey of Autonomous Driving: Common Practices and Emerging Technologies](https://mlsanity.com/paper/1906.05113/) (2019) - [Safety-Enhanced Autonomous Driving Using Interpretable Sensor Fusion Transformer](https://mlsanity.com/paper/2207.14024/) (2022) - [NeuRAD: Neural Rendering for Autonomous Driving](https://mlsanity.com/paper/2311.15260/) (2023) - [Understanding Domain Randomization for Sim-to-real Transfer](https://mlsanity.com/paper/2110.03239/) (2021) - [On Offline Evaluation of Vision-based Driving Models](https://mlsanity.com/paper/1809.04843/) (2018) - [How Simulation Helps Autonomous Driving:A Survey of Sim2real, Digital Twins, and Parallel Intelligence](https://mlsanity.com/paper/2305.01263/) (2023) - [Misbehaviour Prediction for Autonomous Driving Systems](https://mlsanity.com/paper/1910.04443/) (2019) - [Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysis](https://mlsanity.com/paper/2506.11526/) (2025) - [Mind the Gap! A Study on the Transferability of Virtual vs Physical-world Testing of Autonomous Driving Systems](https://mlsanity.com/paper/2112.11255/) (2021) - [A Survey on Automated Driving System Testing: Landscapes and Trends](https://mlsanity.com/paper/2206.05961/) (2022) - [Pylot: A Modular Platform for Exploring Latency-Accuracy Tradeoffs in Autonomous Vehicles](https://mlsanity.com/paper/2104.07830/) (2021) - [Comparing Offline and Online Testing of Deep Neural Networks: An Autonomous Car Case Study](https://mlsanity.com/paper/1912.00805/) (2019) - [Can Offline Testing of Deep Neural Networks Replace Their Online Testing?](https://mlsanity.com/paper/2101.11118/) (2021) - [Are NeRFs ready for autonomous driving? Towards closing the real-to-simulation gap](https://mlsanity.com/paper/2403.16092/) (2024) - [Assessing Quality Metrics for Neural Reality Gap Input Mitigation in Autonomous Driving Testing](https://mlsanity.com/paper/2404.18577/) (2024) - [DeepRacer: Educational Autonomous Racing Platform for Experimentation with Sim2Real Reinforcement Learning](https://mlsanity.com/paper/1911.01562/) (2019) - [DeepPicar: A Low-cost Deep Neural Network-based Autonomous Car](https://mlsanity.com/paper/1712.08644/) (2017) - [Reality Bites: Assessing the Realism of Driving Scenarios with Large Language Models](https://mlsanity.com/paper/2403.09906/) (2024) - [Predicting Safety Misbehaviours in Autonomous Driving Systems using Uncertainty Quantification](https://mlsanity.com/paper/2404.18573/) (2024) - [Efficient Domain Augmentation for Autonomous Driving Testing Using Diffusion Models](https://mlsanity.com/paper/2409.13661/) (2024) - [Two is Better Than One: Digital Siblings to Improve Autonomous Driving Testing](https://mlsanity.com/paper/2305.08060/) (2023) - [A Multi-Layered Approach for Measuring the Simulation-to-Reality Gap of Radar Perception for Autonomous Driving](https://mlsanity.com/paper/2106.08372/) (2021) - [Sim-on-Wheels: Physical World in the Loop Simulation for Self-Driving](https://mlsanity.com/paper/2306.08807/) (2023) - [Search-based DNN Testing and Retraining with GAN-enhanced Simulations](https://mlsanity.com/paper/2406.13359/) (2024) - [Robotics Software Engineering: A Perspective from the Service Robotics Domain](https://mlsanity.com/paper/2006.10608/) (2020) ## Built on by (most cited first, 1 of 1) - [Latent Regularization in Generative Test Input Generation](https://mlsanity.com/paper/2602.15552/) (2026) ## Abstract (arXiv) Simulation-based testing is a cornerstone of Autonomous Driving System (ADS) development, offering safe and scalable evaluation across diverse driving scenarios. However, discrepancies between simulated and real-world behavior, known as the reality gap, challenge the transferability of test results to deployed systems. In this paper, we present a comprehensive empirical study comparing four representative testing modalities: Software-in-the-Loop (SiL), Vehicle-in-the-Loop (ViL), Mixed-Reality (MR), and full real-world testing. Using a small-scale physical vehicle equipped with real sensors (camera and LiDAR) and its digital twin, we implement each setup and evaluate two ADS architectures (modular and end-to-end) across diverse indoor driving scenarios involving real obstacles, road topologies, and indoor environments. We systematically assess the impact of each testing modality along three dimensions of the reality gap: actuation, perception, and behavioral fidelity. Our results show that while SiL and ViL setups simplify critical aspects of real-world dynamics and sensing, MR testing improves perceptual realism without compromising safety or control. Importantly, we identify the conditions under which failures do not transfer across testing modalities and isolate the underlying dimensions of the gap responsible for these discrepancies. Our findings offer actionable insights into the respective strengths and limitations of each modality and outline a path toward more robust and transferable validation of autonomous driving systems.