# DriVerse: Navigation World Model for Driving Simulation via Multimodal Trajectory Prompting and Motion Alignment arXiv: https://arxiv.org/abs/2504.18576 ยท HTML: https://mlsanity.com/paper/2504.18576/ ## Key facts - Cited by 16 papers in the map (data through 14 Oct 2026). - Placed in World models for autonomous driving, one of 11 directions in World models. - Builds on 41 of its 85 references that are in the map. - First posted on arXiv 22 Apr 2025. ## Placed in - World models / [World models for autonomous driving](https://mlsanity.com/wm/world-models-for-autonomous-driving/) - End-to-end autonomous driving / [Generative simulation for driving](https://mlsanity.com/driving/generative-simulation-for-driving/) ## Builds on (most cited first, 30 of 41) - [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020) - [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022) - [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025) - [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023) - [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022) - [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024) - [World Models](https://mlsanity.com/paper/1803.10122/) (2018) - [WorldSimBench: Towards Video Generation Models as World Simulators](https://mlsanity.com/paper/2410.18072/) (2024) - [Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators](https://mlsanity.com/paper/2303.13439/) (2023) - [GAIA-1: A Generative World Model for Autonomous Driving](https://mlsanity.com/paper/2309.17080/) (2023) - [Structure and Content-Guided Video Synthesis with Diffusion Models](https://mlsanity.com/paper/2302.03011/) (2023) - [Latte: Latent Diffusion Transformer for Video Generation](https://mlsanity.com/paper/2401.03048/) (2024) - [VideoCrafter1: Open Diffusion Models for High-Quality Video Generation](https://mlsanity.com/paper/2310.19512/) (2023) - [I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Models](https://mlsanity.com/paper/2311.04145/) (2023) - [Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability](https://mlsanity.com/paper/2405.17398/) (2024) - [MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model](https://mlsanity.com/paper/2311.16498/) (2023) - [Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving](https://mlsanity.com/paper/2311.17918/) (2023) - [Photorealistic Video Generation with Diffusion Models](https://mlsanity.com/paper/2312.06662/) (2023) - [DriveDreamer: Towards Real-world-driven World Models for Autonomous Driving](https://mlsanity.com/paper/2309.09777/) (2023) - [SEINE: Short-to-Long Video Diffusion Model for Generative Transition and Prediction](https://mlsanity.com/paper/2310.20700/) (2023) - [OccWorld: Learning a 3D Occupancy World Model for Autonomous Driving](https://mlsanity.com/paper/2311.16038/) (2023) - [MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation](https://mlsanity.com/paper/2212.09478/) (2022) - [SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Models](https://mlsanity.com/paper/2311.16933/) (2023) - [GenAD: Generalized Predictive Model for Autonomous Driving](https://mlsanity.com/paper/2403.09630/) (2024) - [DriveDreamer4D: World Models Are Effective Data Machines for 4D Driving Scene Representation](https://mlsanity.com/paper/2410.13571/) (2024) - [ADriver-I: A General World Model for Autonomous Driving](https://mlsanity.com/paper/2311.13549/) (2023) - [Motion-Conditioned Diffusion Model for Controllable Video Synthesis](https://mlsanity.com/paper/2304.14404/) (2023) - [DrivingGPT: Unifying Driving World Modeling and Planning with Multi-modal Autoregressive Transformers](https://mlsanity.com/paper/2412.18607/) (2024) - [DriveWorld: 4D Pre-trained Scene Understanding via World Models for Autonomous Driving](https://mlsanity.com/paper/2405.04390/) (2024) - [World Models for Autonomous Driving: An Initial Survey](https://mlsanity.com/paper/2403.02622/) (2024) ## Built on by (most cited first, 16 of 16) - [Controllable Video Generation: A Survey](https://mlsanity.com/paper/2507.16869/) (2025) - [A Comprehensive Survey on World Models for Embodied AI](https://mlsanity.com/paper/2510.16732/) (2025) - [3D and 4D World Modeling: A Survey](https://mlsanity.com/paper/2509.07996/) (2025) - [DrivingGen: A Comprehensive Benchmark for Generative Video World Models in Autonomous Driving](https://mlsanity.com/paper/2601.01528/) (2026) - [Bridging Scene Generation and Planning: Driving with World Model via Unifying Vision and Motion Representation](https://mlsanity.com/paper/2603.14948/) (2026) - [U-ViLAR: Uncertainty-Aware Visual Localization for Autonomous Driving via Differentiable Association and Registration](https://mlsanity.com/paper/2507.04503/) (2025) - [From Instruction to Event: Sound-Triggered Mobile Manipulation](https://mlsanity.com/paper/2601.21667/) (2026) - [FaithFusion: Harmonizing Reconstruction and Generation via Pixel-wise Information Gain](https://mlsanity.com/paper/2511.21113/) (2025) - [Multi-Granularity Reasoning for Image Quality Assessment via Attribute-Aware Reinforcement Learning to Rank](https://mlsanity.com/paper/2604.09704/) (2026) - [SEGAR: Selective Enhancement for Generative Augmented Reality](https://mlsanity.com/paper/2603.24541/) (2026) - [When Numbers Speak: Aligning Textual Numerals and Visual Instances in Text-to-Video Diffusion Models](https://mlsanity.com/paper/2604.08546/) (2026) - [Hierarchical Retrieval Augmented Generation for Adversarial Technique Annotation in Cyber Threat Intelligence Text](https://mlsanity.com/paper/2604.14166/) (2026) - [Avoiding Overthinking and Underthinking: Curriculum-Aware Budget Scheduling for LLMs](https://mlsanity.com/paper/2604.19780/) (2026) - [Taming Visual Neglect: A Variational Information Bottleneck Framework for Adaptive Attention in Multimodal In-Context Learning](https://mlsanity.com/paper/2608.23570/) (2026) - [CST-WM: A Causally Structured World Model for Embodied Visual Tracking](https://mlsanity.com/paper/2609.06302/) (2026) - [TraceICL: Thread-Resolved Demonstration Curation with Consistency Calibration for Robust Multimodal In-Context Learning](https://mlsanity.com/paper/2610.12516/) (2026) ## Abstract (arXiv) This paper presents DriVerse, a generative model for simulating navigation-driven driving scenes from a single image and a future trajectory. Previous autonomous driving world models either directly feed the trajectory or discrete control signals into the generation pipeline, leading to poor alignment between the control inputs and the implicit features of the 2D base generative model, which results in low-fidelity video outputs. Some methods use coarse textual commands or discrete vehicle control signals, which lack the precision to guide fine-grained, trajectory-specific video generation, making them unsuitable for evaluating actual autonomous driving algorithms. DriVerse introduces explicit trajectory guidance in two complementary forms: it tokenizes trajectories into textual prompts using a predefined trend vocabulary for seamless language integration, and converts 3D trajectories into 2D spatial motion priors to enhance control over static content within the driving scene. To better handle dynamic objects, we further introduce a lightweight motion alignment module, which focuses on the inter-frame consistency of dynamic pixels, significantly enhancing the temporal coherence of moving elements over long sequences. With minimal training and no need for additional data, DriVerse outperforms specialized models on future video generation tasks across both the nuScenes and Waymo datasets. The code and models will be released to the public.