# Learning to Drive from Simulation without Real World Labels

arXiv: https://arxiv.org/abs/1812.03823 · HTML: https://mlsanity.com/paper/1812.03823/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Dec 2018.

## Built on by (most cited first, 30 of 39)

- [NuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles](https://mlsanity.com/paper/2106.11810/) (2021)
- [End-to-end Autonomous Driving: Challenges and Frontiers](https://mlsanity.com/paper/2306.16927/) (2023)
- [AdvSim: Generating Safety-Critical Scenarios for Self-Driving Vehicles](https://mlsanity.com/paper/2101.06549/) (2021)
- [Recent Advancements in End-to-End Autonomous Driving using Deep Learning: A Survey](https://mlsanity.com/paper/2307.04370/) (2023)
- [Urban Driving with Conditional Imitation Learning](https://mlsanity.com/paper/1912.00177/) (2019)
- [A Survey of Deep RL and IL for Autonomous Driving Policy Learning](https://mlsanity.com/paper/2101.01993/) (2021)
- [When to Trust Your Simulator: Dynamics-Aware Hybrid Offline-and-Online Reinforcement Learning](https://mlsanity.com/paper/2206.13464/) (2022)
- [Exploring Imitation Learning for Autonomous Driving with Feedback Synthesizer and Differentiable Rasterization](https://mlsanity.com/paper/2103.01882/) (2021)
- [Sim-to-Real Domain Adaptation for Lane Detection and Classification in Autonomous Driving](https://mlsanity.com/paper/2202.07133/) (2022)
- [Learning Interactive Driving Policies via Data-driven Simulation](https://mlsanity.com/paper/2111.12137/) (2021)
- [A Comprehensive Survey of Cross-Domain Policy Transfer for Embodied Agents](https://mlsanity.com/paper/2402.04580/) (2024)
- [Closing the Planning-Learning Loop with Application to Autonomous Driving](https://mlsanity.com/paper/2101.03834/) (2021)
- [Towards Optimal Strategies for Training Self-Driving Perception Models in Simulation](https://mlsanity.com/paper/2111.07971/) (2021)
- [FedRAD: Federated Robust Adaptive Distillation](https://mlsanity.com/paper/2112.01405/) (2021)
- [Two is Better Than One: Digital Siblings to Improve Autonomous Driving Testing](https://mlsanity.com/paper/2305.08060/) (2023)
- [xTED: Cross-Domain Adaptation via Diffusion-Based Trajectory Editing](https://mlsanity.com/paper/2409.08687/) (2024)
- [Autonomous Driving at Unsignalized Intersections: A Review of Decision-Making Challenges and Reinforcement Learning-Based Solutions](https://mlsanity.com/paper/2409.13144/) (2024)
- [End-to-End Intersection Handling using Multi-Agent Deep Reinforcement Learning](https://mlsanity.com/paper/2104.13617/) (2021)
- [A Versatile and Efficient Reinforcement Learning Framework for Autonomous Driving](https://mlsanity.com/paper/2110.11573/) (2021)
- [Sim2Real Diffusion: Leveraging Foundation Vision Language Models for Adaptive Automated Driving](https://mlsanity.com/paper/2507.00236/) (2025)
- [Surround-View Fisheye Optics in Computer Vision and Simulation: Survey and Challenges](https://mlsanity.com/paper/2402.12041/) (2024)
- [No More Blind Spots: Learning Vision-Based Omnidirectional Bipedal Locomotion for Challenging Terrain](https://mlsanity.com/paper/2508.11929/) (2025)
- [On the Choice of Data for Efficient Training and Validation of End-to-End Driving Models](https://mlsanity.com/paper/2206.00608/) (2022)
- [Imitation Learning for Generalizable Self-driving Policy with Sim-to-real Transfer](https://mlsanity.com/paper/2206.10797/) (2022)
- [Controlling Steering with Energy-Based Models](https://mlsanity.com/paper/2301.12264/) (2023)
- [Parallel bootstrap-based on-policy deep reinforcement learning for continuous flow control applications](https://mlsanity.com/paper/2304.12330/) (2023)
- [Combating the effects of speed and delays in end-to-end self-driving](https://mlsanity.com/paper/2312.06670/) (2023)
- [End-to-End Steering for Autonomous Vehicles via Conditional Imitation Co-Learning](https://mlsanity.com/paper/2411.16131/) (2024)
- [Perception Imitation: Towards Synthesis-free Simulator for Autonomous Vehicles](https://mlsanity.com/paper/2304.09365/) (2023)
- [Attacking Motion Planners Using Adversarial Perception Errors](https://mlsanity.com/paper/2311.12722/) (2023)

## Abstract (arXiv)

Simulation can be a powerful tool for understanding machine learning systems and designing methods to solve real-world problems. Training and evaluating methods purely in simulation is often "doomed to succeed" at the desired task in a simulated environment, but the resulting models are incapable of operation in the real world. Here we present and evaluate a method for transferring a vision-based lane following driving policy from simulation to operation on a rural road without any real-world labels. Our approach leverages recent advances in image-to-image translation to achieve domain transfer while jointly learning a single-camera control policy from simulation control labels. We assess the driving performance of this method using both open-loop regression metrics, and closed-loop performance operating an autonomous vehicle on rural and urban roads.