# Real-to-Virtual Domain Unification for End-to-End Autonomous Driving

arXiv: https://arxiv.org/abs/1801.03458 · HTML: https://mlsanity.com/paper/1801.03458/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Jan 2018.

## Built on by (most cited first, 10 of 10)

- [Model-Based Domain Generalization](https://mlsanity.com/paper/2102.11436/) (2021)
- [Progressive Random Convolutions for Single Domain Generalization](https://mlsanity.com/paper/2304.00424/) (2023)
- [Improving Nighttime Driving-Scene Segmentation via Dual Image-adaptive Learnable Filters](https://mlsanity.com/paper/2207.01331/) (2022)
- [MRFP: Learning Generalizable Semantic Segmentation from Sim-2-Real with Multi-Resolution Feature Perturbation](https://mlsanity.com/paper/2311.18331/) (2023)
- [A Bayesian Approach to Reinforcement Learning of Vision-Based Vehicular Control](https://mlsanity.com/paper/2104.03807/) (2021)
- [An Exploratory Study of AI System Risk Assessment from the Lens of Data Distribution and Uncertainty](https://mlsanity.com/paper/2212.06828/) (2022)
- [On the Choice of Data for Efficient Training and Validation of End-to-End Driving Models](https://mlsanity.com/paper/2206.00608/) (2022)
- [Cross-domain error minimization for unsupervised domain adaptation](https://mlsanity.com/paper/2106.15057/) (2021)
- [Imitation Learning Approach for AI Driving Olympics Trained on Real-world and Simulation Data Simultaneously](https://mlsanity.com/paper/2007.03514/) (2020)
- [Unsupervised Super-Resolution of Satellite Imagery for High Fidelity Material Label Transfer](https://mlsanity.com/paper/2105.07322/) (2021)

## Abstract (arXiv)

In the spectrum of vision-based autonomous driving, vanilla end-to-end models are not interpretable and suboptimal in performance, while mediated perception models require additional intermediate representations such as segmentation masks or detection bounding boxes, whose annotation can be prohibitively expensive as we move to a larger scale. More critically, all prior works fail to deal with the notorious domain shift if we were to merge data collected from different sources, which greatly hinders the model generalization ability. In this work, we address the above limitations by taking advantage of virtual data collected from driving simulators, and present DU-drive, an unsupervised real-to-virtual domain unification framework for end-to-end autonomous driving. It first transforms real driving data to its less complex counterpart in the virtual domain and then predicts vehicle control commands from the generated virtual image. Our framework has three unique advantages: 1) it maps driving data collected from a variety of source distributions into a unified domain, effectively eliminating domain shift; 2) the learned virtual representation is simpler than the input real image and closer in form to the "minimum sufficient statistic" for the prediction task, which relieves the burden of the compression phase while optimizing the information bottleneck tradeoff and leads to superior prediction performance; 3) it takes advantage of annotated virtual data which is unlimited and free to obtain. Extensive experiments on two public driving datasets and two driving simulators demonstrate the performance superiority and interpretive capability of DU-drive.