mlsanityData to 7 Oct 2026

Model-Based Imitation Learning for Urban Driving

arXiv 2210.07729 ↗First posted 14 Oct 2022cs.CV cs.AI cs.ROcited by 149

On the map: End-to-end autonomous driving → Imitation-learning end-to-end planners

also in 1 more
Abstract (arXiv)

An accurate model of the environment and the dynamic agents acting in it offers great potential for improving motion planning. We present MILE: a Model-based Imitation LEarning approach to jointly learn a model of the world and a policy for autonomous driving. Our method leverages 3D geometry as an inductive bias and learns a highly compact latent space directly from high-resolution videos of expert demonstrations. Our model is trained on an offline corpus of urban driving data, without any online interaction with the environment. MILE improves upon prior state-of-the-art by 31% in driving score on the CARLA simulator when deployed in a completely new town and new weather conditions. Our model can predict diverse and plausible states and actions, that can be interpretably decoded to bird's-eye view semantic segmentation. Further, we demonstrate that it can execute complex driving manoeuvres from plans entirely predicted in imagination. Our approach is the first camera-only method that models static scene, dynamic scene, and ego-behaviour in an urban driving environment. The code and model weights are available at https://github.com/wayveai/mile.

Key facts. Cited by 149 papers in the map (data through 7 Oct 2026). Placed in Imitation-learning end-to-end planners, one of 9 directions in End-to-end autonomous driving. Builds on 37 of its 59 references that are in the map. First posted on arXiv 14 Oct 2022.

Builds on

37 of 59 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Learning from All Vehicles22 Mar
  2. 2021End-to-End Urban Driving by Imitating a Reinforcement Learning Coach18 Aug

Other references · 35

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  3. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  4. 2017Rethinking Atrous Convolution for Semantic Image Segmentationcited by 2,154
  5. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  6. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  7. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformerscited by 1,213
  8. 2020Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3Dcited by 1,040
  9. 2020Mastering Atari with Discrete World Modelscited by 990
  10. 2018Recurrent World Models Facilitate Policy Evolutioncited by 761
  11. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  12. 2017End-to-end Driving via Conditional Imitation Learningcited by 479
23 more
  1. 2018ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worstcited by 391
  2. 2021Multi-Modal Fusion Transformer for End-to-End Autonomous Drivingcited by 372
  3. 2019Panoptic-DeepLabcited by 362
  4. 2019Learning by Cheatingcited by 361
  5. 2020TNT: Target-driveN Trajectory Predictioncited by 345
  6. 2019Exploring the Limitations of Behavior Cloning for Autonomous Drivingcited by 343
  7. 2017DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agentscited by 310
  8. 2018Stochastic Video Generation with a Learned Priorcited by 244
  9. 2021FIERY: Future Instance Prediction in Bird's-Eye View from Surround Monocular Camerascited by 239
  10. 2017Stochastic Variational Video Predictioncited by 234
  11. 2019PRECOG: PREdiction Conditioned On Goals in Visual Multi-Agent Settingscited by 178
  12. 2021NEAT: Neural Attention Fields for End-to-End Autonomous Drivingcited by 174
  13. 2019End-to-End Model-Free Reinforcement Learning for Urban Driving using Implicit Affordancescited by 147
  14. 2020PLAS: Latent Action Space for Offline Reinforcement Learningcited by 133
  15. 2022BEVSegFormer: Bird's Eye View Semantic Segmentation From Arbitrary Camera Rigscited by 123
  16. 2021Learning to drive from a world on railscited by 114
  17. 2020Stochastic Latent Residual Video Predictioncited by 106
  18. 2019Urban Driving with Conditional Imitation Learningcited by 84
  19. 2018Deep Imitative Models for Flexible Inference, Planning, and Controlcited by 75
  20. 2019Model-Predictive Policy Learning with Uncertainty Regularization for Driving in Dense Trafficcited by 50
  21. 2021Bird's-Eye-View Panoptic Segmentation Using Monocular Frontal View Imagescited by 48
  22. 2021MobILE: Model-Based Imitation Learning From Observation Alonecited by 44
  23. 2022Separating the World and Ego Models for Self-Drivingcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 30; 2024: 47; 2025: 45; 2026: 25.

149 papers.

Sort:

By area:

  1. 2022Planning-oriented Autonomous Drivingcited by 1,068
  2. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 618
  3. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 485
  4. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 337
  5. 2023Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Drivingcited by 309
  6. 2024VADv2: End-to-End Vectorized Autonomous Driving via Probabilistic Planningcited by 287
  7. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Drivingcited by 258
  8. 2024Bench2Drive: Towards Multi-Ability Benchmarking of Closed-Loop End-To-End Autonomous Drivingcited by 250
  9. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Drivingcited by 233
  10. 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Drivingcited by 231
  11. 2024DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generationcited by 186
  12. 2023Think Twice before Driving: Towards Scalable Decoders for End-to-End Autonomous Drivingcited by 168
38 more
  1. 2025FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Drivingcited by 165
  2. 2024Enhancing End-to-End Autonomous Driving with Latent World Modelcited by 161
  3. 2024GenAD: Generalized Predictive Model for Autonomous Drivingcited by 150
  4. 2023LingoQA: Visual Question Answering for Autonomous Drivingcited by 144
  5. 2023DriveAdapter: Breaking the Coupling Barrier of Perception and Planning in End-to-End Autonomous Drivingcited by 142
  6. 2025OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoningcited by 142
  7. 2025DriveTransformer: Unified Transformer for Scalable End-to-End Autonomous Drivingcited by 125
  8. 2024Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyondcited by 111
  9. 2024ManiGaussian: Dynamic Gaussian Splatting for Multi-task Robotic Manipulationcited by 106
  10. 2023Recent Advancements in End-to-End Autonomous Driving using Deep Learning: A Surveycited by 101
  11. 2024DrivingGPT: Unifying Driving World Modeling and Planning with Multi-modal Autoregressive Transformerscited by 96
  12. 2023Hidden Biases of End-to-End Driving Modelscited by 95
  13. 2024World Models for Autonomous Driving: An Initial Surveycited by 79
  14. 2024GaussianWorld: Gaussian World Model for Streaming 3D Occupancy Predictioncited by 66
  15. 2025A Comprehensive Survey on World Models for Embodied AIcited by 65
  16. 20253D and 4D World Modeling: A Surveycited by 64
  17. 2025A Survey of World Models for Autonomous Drivingcited by 60
  18. 2023RealGen: Retrieval Augmented Generation for Controllable Traffic Scenarioscited by 46
  19. 2025ReSim: Reliable World Simulation for Autonomous Drivingcited by 42
  20. 2026World Model for Robot Learning: A Comprehensive Surveycited by 42
  21. 2025Centaur: Robust End-to-End Autonomous Driving with Test-Time Trainingcited by 41
  22. 2025NavMorph: A Self-Evolving World Model for Vision-and-Language Navigation in Continuous Environmentscited by 41
  23. 2024Solving Motion Planning Tasks with a Scalable Generative Modelcited by 40
  24. 2025RAP: 3D Rasterization Augmented End-to-End Planningcited by 39
  25. 2023Model-Based Runtime Monitoring with Interactive Imitation Learningcited by 37
  26. 2025Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysiscited by 37
  27. 2025GWM: Towards Scalable Gaussian World Models for Robotic Manipulationcited by 37
  28. 2025VaViM and VaVAM: Autonomous Driving through Video Generative Modelingcited by 34
  29. 2023MUVO: A Multimodal Generative World Model for Autonomous Driving with Geometric Representationscited by 33
  30. 2023UniWorld: Autonomous Driving Pre-training via World Modelscited by 32
  31. 2023Real-Time Motion Prediction via Heterogeneous Polyline Transformer with Relative Pose Encodingcited by 32
  32. 2023Prompt, Plan, Perform: LLM-based Humanoid Control via Quantized Imitation Learningcited by 31
  33. 2026DriveWorld-VLA: Unified Latent-Space World Modeling with Vision-Language-Action for Autonomous Drivingcited by 31
  34. 2025The Role of World Models in Shaping Autonomous Driving: A Comprehensive Surveycited by 29
  35. 2024End-to-End Autonomous Driving without Costly Modularization and 3D Manual Annotationcited by 27
  36. 2023Grid-Centric Traffic Scenario Perception for Autonomous Driving: A Comprehensive Reviewcited by 25
  37. 2024DrivingDojo Dataset: Advancing Interactive and Knowledge-Enriched Driving World Modelcited by 25
  38. 2023Machine Learning for Autonomous Vehicle's Trajectory Prediction: A comprehensive survey, Challenges, and Future Research Directionscited by 23

50 most cited of 149 All 149 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2021Urban Driver: Learning to Drive from Real-world Demonstrations Using Policy Gradients
  2. 2021Exploring Imitation Learning for Autonomous Driving with Feedback Synthesizer and Differentiable Rasterization
  3. 2022Hierarchical Model-Based Imitation Learning for Planning in Autonomous Driving
  4. 2022SHAIL: Safety-Aware Hierarchical Adversarial Imitation Learning for Autonomous Driving in Urban Environments
  5. 2026Aligning Microscopic Vehicle and Macroscopic Traffic Statistics: Reconstructing Driving Behavior from Partial Data
  6. 2019Conditional Vehicle Trajectories Prediction in CARLA Urban Environment
  7. 2025Learning to Drive by Imitating Surrounding Vehicles
  8. 2020VTGNet: A Vision-based Trajectory Generation Network for Autonomous Vehicles in Urban Environments