# Bridging the Gap between Real-world and Synthetic Images for Testing Autonomous Driving Systems

arXiv: https://arxiv.org/abs/2408.13950 · HTML: https://mlsanity.com/paper/2408.13950/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 67 references that are in the map.
- First posted on arXiv 25 Aug 2024.

## Builds on (most cited first, 30 of 32)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [DeepXplore: Automated Whitebox Testing of Deep Learning Systems](https://mlsanity.com/paper/1705.06640/) (2017)
- [DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Cars](https://mlsanity.com/paper/1708.08559/) (2017)
- [Machine Learning Testing: Survey, Landscapes and Horizons](https://mlsanity.com/paper/1906.10742/) (2019)
- [Neural Style Transfer: A Review](https://mlsanity.com/paper/1705.04058/) (2017)
- [Deep Photo Style Transfer](https://mlsanity.com/paper/1703.07511/) (2017)
- [Accelerated Hierarchical Density Clustering](https://mlsanity.com/paper/1705.07321/) (2017)
- [How good is my GAN?](https://mlsanity.com/paper/1807.09499/) (2018)
- [Dual Contrastive Learning for Unsupervised Image-to-Image Translation](https://mlsanity.com/paper/2104.07689/) (2021)
- [On Offline Evaluation of Vision-based Driving Models](https://mlsanity.com/paper/1809.04843/) (2018)
- [Misbehaviour Prediction for Autonomous Driving Systems](https://mlsanity.com/paper/1910.04443/) (2019)
- [DeepBillboard: Systematic Physical-World Testing of Autonomous Driving Systems](https://mlsanity.com/paper/1812.10812/) (2018)
- [Model-based Exploration of the Frontier of Behaviours for Deep Learning System Testing](https://mlsanity.com/paper/2007.02787/) (2020)
- [Mind the Gap! A Study on the Transferability of Virtual vs Physical-world Testing of Autonomous Driving Systems](https://mlsanity.com/paper/2112.11255/) (2021)
- [A Survey on Automated Driving System Testing: Landscapes and Trends](https://mlsanity.com/paper/2206.05961/) (2022)
- [Comparing Offline and Online Testing of Deep Neural Networks: An Autonomous Car Case Study](https://mlsanity.com/paper/1912.00805/) (2019)
- [Can Offline Testing of Deep Neural Networks Replace Their Online Testing?](https://mlsanity.com/paper/2101.11118/) (2021)
- [Distribution-Aware Testing of Neural Networks Using Generative Models](https://mlsanity.com/paper/2102.13602/) (2021)
- [Testing of Autonomous Driving Systems: Where Are We and Where Should We Go?](https://mlsanity.com/paper/2106.12233/) (2021)
- [When and Why Test Generators for Deep Learning Produce Invalid Inputs: an Empirical Study](https://mlsanity.com/paper/2212.11368/) (2022)
- [Assessing Quality Metrics for Neural Reality Gap Input Mitigation in Autonomous Driving Testing](https://mlsanity.com/paper/2404.18577/) (2024)
- [Digital Twins Are Not Monozygotic -- Cross-Replicating ADAS Testing in Two Industry-Grade Automotive Simulators](https://mlsanity.com/paper/2012.06822/) (2020)
- [YOLO-ReT: Towards High Accuracy Real-time Object Detection on Edge GPUs](https://mlsanity.com/paper/2110.13713/) (2021)
- [Two is Better Than One: Digital Siblings to Improve Autonomous Driving Testing](https://mlsanity.com/paper/2305.08060/) (2023)
- [Sim2real for Autonomous Vehicle Control using Executable Digital Twin](https://mlsanity.com/paper/2208.06226/) (2022)
- [Evaluating the Impact of Flaky Simulators on Testing Autonomous Driving Systems](https://mlsanity.com/paper/2311.18768/) (2023)
- [Reflections on Surrogate-Assisted Search-Based Testing: A Taxonomy and Two Replication Studies based on Industrial ADAS and Simulink Models](https://mlsanity.com/paper/2305.00083/) (2023)

## Built on by (most cited first, 4 of 4)

- [A Multi-Modality Evaluation of the Reality Gap in Autonomous Driving Systems](https://mlsanity.com/paper/2509.22379/) (2025)
- [CARD: A Multi-Modal Automotive Dataset for Dense 3D Reconstruction in Challenging Road Topography](https://mlsanity.com/paper/2605.05014/) (2026)
- [GAN-enhanced Simulation-driven DNN Testing in Absence of Ground Truth](https://mlsanity.com/paper/2503.15953/) (2025)
- [Effort-Optimized, Accuracy-Driven Labelling and Validation of Test Inputs for DL Systems: A Mixed-Integer Linear Programming Approach](https://mlsanity.com/paper/2507.04990/) (2025)

## Abstract (arXiv)

Deep Neural Networks (DNNs) for Autonomous Driving Systems (ADS) are typically trained on real-world images and tested using synthetic simulator images. This approach results in training and test datasets with dissimilar distributions, which can potentially lead to erroneously decreased test accuracy. To address this issue, the literature suggests applying domain-to-domain translators to test datasets to bring them closer to the training datasets. However, translating images used for testing may unpredictably affect the reliability, effectiveness and efficiency of the testing process. Hence, this paper investigates the following questions in the context of ADS: Could translators reduce the effectiveness of images used for ADS-DNN testing and their ability to reveal faults in ADS-DNNs? Can translators result in excessive time overhead during simulation-based testing? To address these questions, we consider three domain-to-domain translators: CycleGAN and neural style transfer, from the literature, and SAEVAE, our proposed translator. Our results for two critical ADS tasks -- lane keeping and object detection -- indicate that translators significantly narrow the gap in ADS test accuracy caused by distribution dissimilarities between training and test data, with SAEVAE outperforming the other two translators. We show that, based on the recent diversity, coverage, and fault-revealing ability metrics for testing deep-learning systems, translators do not compromise the diversity and the coverage of test data, nor do they lead to revealing fewer faults in ADS-DNNs. Further, among the translators considered, SAEVAE incurs a negligible overhead in simulation time and can be efficiently integrated into simulation-based testing. Finally, we show that translators increase the correlation between offline and simulation-based testing results, which can help reduce the cost of simulation-based testing.