mlsanityData to 7 Oct 2026

Latent World Models for Automated Driving: A Unified Taxonomy, Evaluation Framework, and Open Challenges

arXiv 2603.09086 ↗First posted 10 Mar 2026cs.RO cs.AI cs.LGcited by 3

On the map: World models → World models for autonomous driving

also in 2 more
Abstract (arXiv)

Emerging generative world models and vision-language-action (VLA) systems are rapidly reshaping automated driving by enabling scalable simulation, long-horizon forecasting, and capability-rich decision making. Across these directions, latent representations serve as the central computational substrate: they compress high-dimensional multi-sensor observations, enable temporally coherent rollouts, and provide interfaces for planning, reasoning, and controllable generation. This paper proposes a unifying latent-space framework that synthesizes recent progress in world models for automated driving. The framework organizes the design space by the target and form of latent representations (latent worlds, latent actions, latent generators; continuous states, discrete tokens, and hybrids) and by structural priors for geometry, topology, and semantics. Building on this taxonomy, the paper articulates five cross-cutting internal mechanics (i.e, structural isomorphism, long-horizon temporal stability, semantic and reasoning alignment, value-aligned objectives and post-training, as well as adaptive computation and deliberation) and connects these design choices to robustness, generalization, and deployability. The work also proposes concrete evaluation prescriptions, including a closed-loop metric suite and a resource-aware deliberation cost, designed to reduce the open-loop / closed-loop mismatch. Finally, the paper identifies actionable research directions toward advancing latent world model for decision-ready, verifiable, and resource-efficient automated driving.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in World models for autonomous driving, one of 11 directions in World models. Builds on 73 of its 87 references that are in the map. First posted on arXiv 10 Mar 2026.

Builds on

73 of 87 references.

Closest ancestors · 25

Same direction, newest first

  1. 2025DriveLaW:Unifying Planning and Video Generation in a Latent Driving World29 Dec
  2. 2025WorldRFT: Latent World Model Planning with Reinforcement Fine-Tuning for Autonomous Driving22 Dec
  3. 2025Latent Chain-of-Thought World Modeling for End-to-End Driving11 Dec
  4. 2025MindDrive: An All-in-One Framework Bridging World Models and Vision-Language Model for End-to-End Autonomous Driving4 Dec
  5. 2025AD-R1: Closed-Loop Reinforcement Learning for End-to-End Autonomous Driving with Impartial World Models25 Nov
  6. 2025Orbis: Overcoming Challenges of Long-Horizon Prediction in Driving World Models17 Jul
  7. 2025World4Drive: End-to-End Autonomous Driving via Intention-aware Physical Latent World Model1 Jul
  8. 2025Epona: Autoregressive Diffusion World Model for Autonomous Driving30 Jun
  9. 2025ReSim: Reliable World Simulation for Autonomous Driving11 Jun
  10. 2025LongDWM: Cross-Granularity Distillation for Building a Long-Term Driving World Model2 Jun
  11. 2025DriveX: Omni Scene Modeling for Learning Generalizable World Knowledge in Autonomous Driving25 May
  12. 2025Raw2Drive: Reinforcement Learning with Aligned World Models for End-to-End Autonomous Driving (in CARLA v2)22 May
13 more
  1. 2025GAIA-2: A Controllable Multi-View Generative World Model for Autonomous Driving26 Mar
  2. 2025Unraveling the Effects of Synthetic Data on End-to-End Autonomous Driving23 Mar
  3. 2024Learning Multiple Probabilistic Decisions from Latent World Model in Autonomous Driving24 Sep
  4. 2024Driving in the Occupancy World: Vision-Centric 4D Occupancy Forecasting and Planning via World Models for Autonomous Driving26 Aug
  5. 2024BEVWorld: A Multimodal World Simulator for Autonomous Driving via Scene-Level BEV Latents8 Jul
  6. 2024Enhancing End-to-End Autonomous Driving with Latent World Model12 Jun
  7. 2024DriveWorld: 4D Pre-trained Scene Understanding via World Models for Autonomous Driving7 May
  8. 2024Think2Drive: Efficient Reinforcement Learning by Thinking in Latent World Model for Quasi-Realistic Autonomous Driving (in CARLA-v2)26 Feb
  9. 2024GenAD: Generative End-to-End Autonomous Driving18 Feb
  10. 2023Waymax: An Accelerated, Data-Driven Simulator for Large-Scale Autonomous Driving Research12 Oct
  11. 2022Enhance Sample Efficiency and Robustness of End-to-end Urban Autonomous Driving via Semantic Masked World Model8 Oct
  12. 2021MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning26 Sep
  13. 2017CARLA: An Open Urban Driving Simulator10 Nov

Other references · 48

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  5. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  6. 2022Flow Matching for Generative Modelingcited by 5,426
  7. 2017Neural Discrete Representation Learningcited by 4,650
  8. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  9. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,436
  10. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,335
  11. 2018World Modelscited by 1,444
  12. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
36 more
  1. 2023Mastering Diverse Domains through World Modelscited by 1,193
  2. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
  3. 2017AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehiclescited by 517
  4. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 485
  5. 2017End-to-end Driving via Conditional Imitation Learningcited by 479
  6. 2018ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worstcited by 391
  7. 2025Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Modelscited by 376
  8. 2024NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarkingcited by 343
  9. 2023OpenOccupancy: A Large Scale Benchmark for Surrounding Semantic Occupancy Perceptioncited by 218
  10. 2025FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Drivingcited by 165
  11. 2025Alpamayo-R1: Bridging Reasoning and Action Prediction for Generalizable Autonomous Driving in the Long Tailcited by 136
  12. 2020LGSVL Simulator: A High Fidelity Simulator for Autonomous Drivingcited by 119
  13. 2020Interpretable End-to-end Urban Autonomous Driving with Latent Deep Reinforcement Learningcited by 99
  14. 2024MagicDrive-V2: High-Resolution Long Video Generation for Autonomous Driving with Adaptive Controlcited by 90
  15. 2025Don't Shake the Wheel: Momentum-Aware Planning in End-to-End Autonomous Drivingcited by 84
  16. 2023How Simulation Helps Autonomous Driving:A Survey of Sim2real, Digital Twins, and Parallel Intelligencecited by 55
  17. 2024Can Vehicle Motion Planning Generalize to Realistic Long-tail Scenarios?cited by 38
  18. 2024SubjectDrive: Scaling Generative Data in Autonomous Driving via Subject Controlcited by 32
  19. 2024UniMLVG: Unified Framework for Multi-view Long Video Generation with Comprehensive Control Capabilities for Autonomous Drivingcited by 27
  20. 2025Glad: A Streaming Scene Generator for Autonomous Drivingcited by 17
  21. 2025VERDI: VLM-Embedded Reasoning for Autonomous Drivingcited by 15
  22. 2025CoGen: 3D Consistent Video Generation via Adaptive Conditioning for Autonomous Drivingcited by 12
  23. 2024Simplifying Latent Dynamics with Softly State-Invariant World Modelscited by 10
  24. 2025ColaVLA: Leveraging Cognitive Latent Reasoning for Hierarchical Parallel Trajectory Planning in Autonomous Drivingcited by 10
  25. 2024Curb Your Attention: Causal Attention Gating for Robust Trajectory Prediction in Autonomous Drivingcited by 9
  26. 2025OmniDrive-R1: Reinforcement-driven Interleaved Multi-modal Chain-of-Thought for Trustworthy Vision-Language Autonomous Drivingcited by 9
  27. 2024Efficient and Generalized end-to-end Autonomous Driving System with Latent Deep Reinforcement Learning and Demonstrationscited by 7
  28. 2024Exploring Latent Pathways: Enhancing the Interpretability of Autonomous Driving with a Variational Autoencodercited by 6
  29. 2025OmniGen: Unified Multimodal Sensor Generation for Autonomous Drivingcited by 5
  30. 2024Validity Learning on Failures: Mitigating the Distribution Shift in Autonomous Vehicle Planningcited by 4
  31. 2024LiDAR-EDIT: LiDAR Data Generation by Editing the Object Layouts in Real-World Scenescited by 4
  32. 2025VLA-R: Vision-Language Action Retrieval toward Open-World End-to-End Autonomous Drivingcited by 4
  33. 2023Safe, Efficient, Comfort, and Energy-saving Automated Driving through Roundabout Based on Deep Reinforcement Learningcited by 3
  34. 2025Safety-Critical Traffic Simulation with Guided Latent Diffusion Modelcited by 3
  35. 2023Comprehensive Training and Evaluation on Deep Reinforcement Learning for Automated Driving in Various Simulated Driving Maneuverscited by 2
  36. 2024Towards Understanding Worldwide Cross-cultural Differences in Implicit Driving Cues: Review, Comparative Analysis, and Research Roadmapcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2026LaST-VLA: Thinking in Latent Spatio-Temporal Space for Vision-Language-Action in Autonomous Driving
  2. 2026Latent-WAM: Latent World Action Modeling for End-to-End Autonomous Driving
  3. 2025A Survey of World Models for Autonomous Driving
  4. 2026Kinematics-Aware Latent World Models for Data-Efficient Autonomous Driving
  5. 2026Relationally Grounded Latent World Models for Autonomous Driving
  6. 2026Xiaomi OneVL: One-Step Latent Reasoning and Planning with Vision-Language Explanation
  7. 2026DriveWorld-VLA: Unified Latent-Space World Modeling with Vision-Language-Action for Autonomous Driving
  8. 2026Drive-HWM: Hierarchical World Models for Dynamic-Latent Guided Autonomous Driving