mlsanityData to 14 Oct 2026

Learning to Drive from a World Model

arXiv 2504.19077 ↗First posted 27 Apr 2025cs.CV cs.ROcited by 6

On the map: World models → World models for autonomous driving

also in 1 more
Abstract (arXiv)

Most self-driving systems rely on hand-coded perception outputs and engineered driving rules. Learning directly from human driving data with an end-to-end method can allow for a training architecture that is simpler and scales well with compute and data. In this work, we propose an end-to-end training architecture that uses real driving data to train a driving policy in an on-policy simulator. We show two different methods of simulation, one with reprojective simulation and one with a learned world model. We show that both methods can be used to train a policy that learns driving behavior without any hand-coded driving rules. We evaluate the performance of these policies in a closed-loop simulation and when deployed in a real-world advanced driver-assistance system.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in World models for autonomous driving, one of 11 directions in World models. Builds on 24 of its 33 references that are in the map. First posted on arXiv 27 Apr 2025.

Builds on

24 of 33 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023GAIA-1: A Generative World Model for Autonomous Driving29 Sep
  2. 2021MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning26 Sep
  3. 2018Multimodal Trajectory Predictions for Autonomous Driving using Deep Convolutional Networks18 Sep
  4. 2017CARLA: An Open Urban Driving Simulator10 Nov

Other references · 20

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  4. 2022Scalable Diffusion Models with Transformerscited by 6,054
  5. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  6. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
  7. 2020Shortcut Learning in Deep Neural Networkscited by 1,969
  8. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,498
  9. 2016End to End Learning for Self-Driving Carscited by 1,009
  10. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  11. 2018Recurrent World Models Facilitate Policy Evolutioncited by 785
  12. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 672
8 more
  1. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 345
  2. 2024Diffusion Models Are Real-Time Game Enginescited by 296
  3. 2018Learning to Drive in a Daycited by 273
  4. 2024Navigation World Modelscited by 272
  5. 2016CAD2RL: Real Single-Image Flight without a Single Real Imagecited by 233
  6. 2019Understanding and Improving Layer Normalizationcited by 217
  7. 2023FastViT: A Fast Hybrid Vision Transformer using Structural Reparameterizationcited by 162
  8. 2016Learning a Driving Simulatorcited by 33

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 1.

6 papers.

  1. 2025A Comprehensive Survey on World Models for Embodied AIcited by 68
  2. 2025ReSim: Reliable World Simulation for Autonomous Drivingcited by 44
  3. 2025SimScale: Learning to Drive via Real-World Simulation at Scalecited by 36
  4. 2026Learning Sidewalk Autopilot from Multi-Scale Imitation with Corrective Behavior Expansioncited by 1
  5. 2025Interpretability and Generalization Bounds for Learning Spatial Physicscited by 0
  6. 2025Synthetic Error Injection Fails to Elicit Self-Correction In Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Urban Driver: Learning to Drive from Real-world Demonstrations Using Policy Gradients
  2. 2021SimNet: Learning Reactive Self-driving Simulations from Real-world Observations
  3. 2026Post-Training in End-to-End Autonomous Driving
  4. 2022Learning from All Vehicles
  5. 2026DriveZero: End-to-End Driving Beyond Human Demonstrations
  6. 2022Tackling Real-World Autonomous Driving using Deep Reinforcement Learning
  7. 2022Discrete Control in Real-World Driving Environments using Deep Reinforcement Learning
  8. 2026OPTED: On-Policy Fine-Tuning for End-to-End Driving using a Render-Free Teacher