mlsanityData to 7 Oct 2026

Bridging Scene Generation and Planning: Driving with World Model via Unifying Vision and Motion Representation

arXiv 2603.14948 ↗First posted 16 Mar 2026cs.CVcited by 13

On the map: World models → World models for autonomous driving

also in 1 more
Abstract (arXiv)

End-to-end autonomous driving aims to generate safe and plausible planning policies from raw sensor input. Driving world models have shown great potential in learning rich representations by predicting the future evolution of a driving scene. However, existing driving world models primarily focus on visual scene representation, and motion representation is not explicitly designed to be planner-shared and inheritable, leaving a schism between the optimization of visual scene generation and the requirements of precise motion planning. We present WorldDrive, a holistic framework that couples scene generation and real-time planning via unifying vision and motion representation. We first introduce a Trajectory-aware Driving World Model, which conditions on a trajectory vocabulary to enforce consistency between visual dynamics and motion intentions, enabling the generation of diverse and plausible future scenes conditioned on a specific trajectory. We transfer the vision and motion encoders to a downstream Multi-modal Planner, ensuring the driving policy operates on mature representations pre-optimized by scene generation. A simple interaction between motion representation, visual representation, and ego status can generate high-quality, multi-modal trajectories. Furthermore, to exploit the world model's foresight, we propose a Future-aware Rewarder, which distills future latent representation from the frozen world model to evaluate and select optimal trajectories in real-time. Extensive experiments on the NAVSIM, NAVSIM-v2, and nuScenes benchmarks demonstrate that WorldDrive achieves leading planning performance among vision-only methods while maintaining high-fidelity action-controlled video generation capabilities, providing strong evidence for the effectiveness of unifying vision and motion representation for robust autonomous driving.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in World models for autonomous driving, one of 11 directions in World models. Builds on 57 of its 61 references that are in the map. First posted on arXiv 16 Mar 2026.

Builds on

57 of 61 references.

Closest ancestors · 21

Same direction, newest first

  1. 2025WorldRFT: Latent World Model Planning with Reinforcement Fine-Tuning for Autonomous Driving22 Dec
  2. 2025DriveVLA-W0: World Models Amplify Data Scaling Law in Autonomous Driving14 Oct
  3. 2025World4Drive: End-to-End Autonomous Driving via Intention-aware Physical Latent World Model1 Jul
  4. 2025Epona: Autoregressive Diffusion World Model for Autonomous Driving30 Jun
  5. 2025ReSim: Reliable World Simulation for Autonomous Driving11 Jun
  6. 2025DriVerse: Navigation World Model for Driving Simulation via Multimodal Trajectory Prompting and Motion Alignment22 Apr
  7. 2025End-to-End Driving with Online Trajectory Evaluation via BEV World Model2 Apr
  8. 2025The Role of World Models in Shaping Autonomous Driving: A Comprehensive Survey14 Feb
  9. 2025A Survey of World Models for Autonomous Driving20 Jan
  10. 2024DrivingGPT: Unifying Driving World Modeling and Planning with Multi-modal Autoregressive Transformers24 Dec
  11. 2024InfinityDrive: Breaking Time Limits in Driving World Models2 Dec
  12. 2024Enhancing End-to-End Autonomous Driving with Latent World Model12 Jun
9 more
  1. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability27 May
  2. 2024DriveWorld: 4D Pre-trained Scene Understanding via World Models for Autonomous Driving7 May
  3. 2024GenAD: Generalized Predictive Model for Autonomous Driving14 Mar
  4. 2024World Models for Autonomous Driving: An Initial Survey5 Mar
  5. 2023Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving29 Nov
  6. 2023OccWorld: Learning a 3D Occupancy World Model for Autonomous Driving27 Nov
  7. 2023ADriver-I: A General World Model for Autonomous Driving22 Nov
  8. 2023GAIA-1: A Generative World Model for Autonomous Driving29 Sep
  9. 2021NuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles22 Jun

Other references · 36

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  3. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,198
  4. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
  5. 2022Planning-oriented Autonomous Drivingcited by 1,068
  6. 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 613
  7. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 485
  8. 2022TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Drivingcited by 477
  9. 2024NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarkingcited by 343
  10. 2024DiffusionDrive: Truncated Diffusion Model for End-to-End Autonomous Drivingcited by 324
  11. 2024VADv2: End-to-End Vectorized Autonomous Driving via Probabilistic Planningcited by 287
  12. 2024Hydra-MDP: End-to-end Multimodal Planning with Multi-target Hydra-Distillationcited by 244
24 more
  1. 2023Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving?cited by 242
  2. 2024SparseDrive: End-to-End Autonomous Driving via Sparse Scene Representationcited by 239
  3. 2025FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Drivingcited by 165
  4. 2025Pseudo-Simulation for Autonomous Drivingcited by 141
  5. 2025DriveTransformer: Unified Transformer for Scalable End-to-End Autonomous Drivingcited by 125
  6. 2023Motion Planning for Autonomous Driving: The State of the Art and Future Perspectivescited by 112
  7. 2024UniScene: Unified Occupancy-centric Driving Scene Generationcited by 102
  8. 2023Recent Advancements in End-to-End Autonomous Driving using Deep Learning: A Surveycited by 101
  9. 2023WoVoGen: World Volume-aware Diffusion for Controllable Multi-camera Driving Scene Generationcited by 93
  10. 2024MagicDrive-V2: High-Resolution Long Video Generation for Autonomous Driving with Adaptive Controlcited by 90
  11. 2025Cosmos-Drive-Dreams: Scalable Synthetic Driving Data Generation with World Foundation Modelscited by 80
  12. 2024GEM: A Generalizable Ego-Vision Multimodal World Model for Fine-Grained Ego-Motion, Object Dynamics, and Scene Composition Controlcited by 76
  13. 2023FusionAD: Multi-modality Fusion for Prediction and Planning Tasks of Autonomous Drivingcited by 67
  14. 20253D and 4D World Modeling: A Surveycited by 64
  15. 2024SparseAD: Sparse Query-Centric Paradigm for Efficient End-to-End Autonomous Drivingcited by 46
  16. 2024DrivingSphere: Building a High-fidelity 4D World for Closed-loop Simulationcited by 34
  17. 2025Vision-Language-Action Models for Autonomous Driving: Past, Present, and Futurecited by 32
  18. 2023The Integration of Prediction and Planning in Deep Learning Automated Driving Systems: A Reviewcited by 29
  19. 2025From Forecasting to Planning: Policy World Model for Collaborative State-Action Predictioncited by 29
  20. 2025ImagiDrive: A Unified Imagination-and-Planning Framework for Autonomous Drivingcited by 24
  21. 2025WorldLens: Full-Spectrum Evaluations of Driving World Models in Real Worldcited by 22
  22. 2025TrajDiff: End-to-end Autonomous Driving without Perception Annotationcited by 8
  23. 2025TeraSim: Uncovering Unknown Unsafe Events for Autonomous Vehicles through Generative Simulationcited by 6
  24. 2025TeraSim-World: Worldwide Safety-Critical Data Synthesis for End-to-End Autonomous Drivingcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 13.

13 papers.

Sort:

By area:

Similar, not cited

Close in wording, no citation either way

  1. 2026UniDrive-WM: Unified Understanding, Planning and Generation World Model for Autonomous Driving
  2. 2026ReDrive: Shaping Representations with World Modeling for End-to-End Driving
  3. 2026UniDWM: Towards a Unified Driving World Model via Multifaceted Representation Learning
  4. 2025DriveLaW:Unifying Planning and Video Generation in a Latent Driving World
  5. 2026Uni-World VLA: Interleaved World Modeling and Planning for Autonomous Driving
  6. 2024Driving in the Occupancy World: Vision-Centric 4D Occupancy Forecasting and Planning via World Models for Autonomous Driving
  7. 2026Drive-HWM: Hierarchical World Models for Dynamic-Latent Guided Autonomous Driving
  8. 2026Learning Vision-Language-Action World Models for Autonomous Driving