# Driving Policy Transfer via Modularity and Abstraction

arXiv: https://arxiv.org/abs/1804.09364 · HTML: https://mlsanity.com/paper/1804.09364/

## Key facts

- Cited by 92 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Apr 2018.

## Built on by (most cited first, 30 of 92)

- [End-to-end Interpretable Neural Motion Planner](https://mlsanity.com/paper/2101.06679/) (2021)
- [MP3: A Unified Model to Map, Perceive, Predict and Plan](https://mlsanity.com/paper/2101.06806/) (2021)
- [End-to-End Urban Driving by Imitating a Reinforcement Learning Coach](https://mlsanity.com/paper/2108.08265/) (2021)
- [NEAT: Neural Attention Fields for End-to-End Autonomous Driving](https://mlsanity.com/paper/2109.04456/) (2021)
- [AdvSim: Generating Safety-Critical Scenarios for Self-Driving Vehicles](https://mlsanity.com/paper/2101.06549/) (2021)
- [Learning High-Speed Flight in the Wild](https://mlsanity.com/paper/2110.05113/) (2021)
- [Learning to drive from a world on rails](https://mlsanity.com/paper/2105.00636/) (2021)
- [Navigating to Objects in the Real World](https://mlsanity.com/paper/2212.00922/) (2022)
- [PlanT: Explainable Planning Transformers via Object-Level Representations](https://mlsanity.com/paper/2210.14222/) (2022)
- [Towards learning-based planning:The nuPlan benchmark for real-world autonomous driving](https://mlsanity.com/paper/2403.04133/) (2024)
- [LookOut: Diverse Multi-Future Prediction and Planning for Self-Driving](https://mlsanity.com/paper/2101.06547/) (2021)
- [Urban Driving with Conditional Imitation Learning](https://mlsanity.com/paper/1912.00177/) (2019)
- [Explainability of deep vision-based autonomous driving systems: Review and challenges](https://mlsanity.com/paper/2101.05307/) (2021)
- [ADAPT: Action-aware Driving Caption Transformer](https://mlsanity.com/paper/2302.00673/) (2023)
- [Zero Experience Required: Plug & Play Modular Transfer Learning for Semantic Visual Navigation](https://mlsanity.com/paper/2202.02440/) (2022)
- [Robust Autonomy Emerges from Self-Play](https://mlsanity.com/paper/2502.03349/) (2025)
- [When Physics Meets Machine Learning: A Survey of Physics-Informed Machine Learning](https://mlsanity.com/paper/2203.16797/) (2022)
- [A Survey of Deep RL and IL for Autonomous Driving Policy Learning](https://mlsanity.com/paper/2101.01993/) (2021)
- [Drive Anywhere: Generalizable End-to-end Autonomous Driving with Multi-modal Foundation Models](https://mlsanity.com/paper/2310.17642/) (2023)
- [Semantics for Robotic Mapping, Perception and Interaction: A Survey](https://mlsanity.com/paper/2101.00443/) (2021)
- [Hierarchical Representations and Explicit Memory: Learning Effective Navigation Policies on 3D Scene Graphs using Graph Neural Networks](https://mlsanity.com/paper/2108.01176/) (2021)
- [Learning Robust Autonomous Navigation and Locomotion for Wheeled-Legged Robots](https://mlsanity.com/paper/2405.01792/) (2024)
- [Hierarchical Model-Based Imitation Learning for Planning in Autonomous Driving](https://mlsanity.com/paper/2210.09539/) (2022)
- [Learning a State Representation and Navigation in Cluttered and Dynamic Environments](https://mlsanity.com/paper/2103.04351/) (2021)
- [The Integration of Prediction and Planning in Deep Learning Automated Driving Systems: A Review](https://mlsanity.com/paper/2308.05731/) (2023)
- [GINA-3D: Learning to Generate Implicit Neural Assets in the Wild](https://mlsanity.com/paper/2304.02163/) (2023)
- [The Reality Gap in Robotics: Challenges, Solutions, and Best Practices](https://mlsanity.com/paper/2510.20808/) (2025)
- [Learning by Watching](https://mlsanity.com/paper/2106.05966/) (2021)
- [Natural Language Can Help Bridge the Sim2Real Gap](https://mlsanity.com/paper/2405.10020/) (2024)
- [StyleDrive: Towards Driving-Style Aware Benchmarking of End-To-End Autonomous Driving](https://mlsanity.com/paper/2506.23982/) (2025)

## Abstract (arXiv)

End-to-end approaches to autonomous driving have high sample complexity and are difficult to scale to realistic urban driving. Simulation can help end-to-end driving systems by providing a cheap, safe, and diverse training environment. Yet training driving policies in simulation brings up the problem of transferring such policies to the real world. We present an approach to transferring driving policies from simulation to reality via modularity and abstraction. Our approach is inspired by classic driving systems and aims to combine the benefits of modular architectures and end-to-end deep learning approaches. The key idea is to encapsulate the driving policy such that it is not directly exposed to raw perceptual input or low-level vehicle dynamics. We evaluate the presented approach in simulated urban environments and in the real world. In particular, we transfer a driving policy trained in simulation to a 1/5-scale robotic truck that is deployed in a variety of conditions, with no finetuning, on two continents. The supplementary video can be viewed at https://youtu.be/BrMDJqI6H5U