UniDriveDreamer: A Single-Stage Multimodal World Model for Autonomous Driving
On the map: World models → World models for autonomous driving
also in 1 more
Abstract (arXiv)
World models have demonstrated significant promise for data synthesis in autonomous driving. However, existing methods predominantly concentrate on single-modality generation, typically focusing on either multi-camera video or LiDAR sequence synthesis. In this paper, we propose UniDriveDreamer, a single-stage unified multimodal world model for autonomous driving, which directly generates multimodal future observations without relying on intermediate representations or cascaded modules. Our framework introduces a LiDAR-specific variational autoencoder (VAE) designed to encode input LiDAR sequences, alongside a video VAE for multi-camera images. To ensure cross-modal compatibility and training stability, we propose Unified Latent Anchoring (ULA), which explicitly aligns the latent distributions of the two modalities. The aligned features are fused and processed by a diffusion transformer that jointly models their geometric correspondence and temporal evolution. Additionally, structured scene layout information is projected per modality as a conditioning signal to guide the synthesis. Extensive experiments demonstrate that UniDriveDreamer outperforms previous state-of-the-art methods in both video and LiDAR generation, while also yielding measurable improvements in downstream
Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in World models for autonomous driving, one of 11 directions in World models. Builds on 45 of its 53 references that are in the map. First posted on arXiv 2 Feb 2026.
Builds on
45 of 53 references.
Closest ancestors · 10
Same direction, newest first
- 2025GAIA-2: A Controllable Multi-View Generative World Model for Autonomous Driving26 Mar
- 2025ReconDreamer++: Harmonizing Generative and Reconstructive Models for Driving Scene Representation24 Mar
- 2024ReconDreamer: Crafting World Models for Driving Scene Reconstruction via Online Restoration29 Nov
- 2024DriveDreamer4D: World Models Are Effective Data Machines for 4D Driving Scene Representation17 Oct
- 2024Unleashing Generalization of End-to-End Autonomous Driving with Controllable Long Video Generation3 Jun
- 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability27 May
- 2024GenAD: Generalized Predictive Model for Autonomous Driving14 Mar
- 2024DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generation11 Mar
- 2023Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving29 Nov
- 2023GAIA-1: A Generative World Model for Autonomous Driving29 Sep
Other references · 35
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
- 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
- 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
- 2022Flow Matching for Generative Modelingcited by 5,426
- 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
- 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
- 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,536
- 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
- 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,597
- 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
- 2022BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representationcited by 844
- 2025Cosmos World Foundation Model Platform for Physical AIcited by 833
23 more
- 2023MagicDrive: Street View Generation with Diverse 3D Geometry Controlcited by 250
- 2025World Simulation with Video Foundation Models for Physical AIcited by 183
- 2023Panacea: Panoramic and Controllable Video Generation for Autonomous Drivingcited by 165
- 2023UniMax: Fairer and more Effective Language Sampling for Large-Scale Multilingual Pretrainingcited by 147
- 2024Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyondcited by 111
- 2023Street-View Image Generation from a Bird's-Eye View Layoutcited by 102
- 2023BEVControl: Accurately Controlling Street-view Elements with Multi-perspective Consistency via BEV Sketch Layoutcited by 102
- 2024UniScene: Unified Occupancy-centric Driving Scene Generationcited by 102
- 2025Cosmos-Transfer1: Conditional World Generation with Adaptive Multimodal Controlcited by 95
- 2022Learning to Generate Realistic LiDAR Point Cloudscited by 91
- 2024MagicDrive-V2: High-Resolution Long Video Generation for Autonomous Driving with Adaptive Controlcited by 90
- 2024WorldDreamer: Towards General World Models for Video Generation via Predicting Masked Tokenscited by 87
- 2025Cosmos-Drive-Dreams: Scalable Synthetic Driving Data Generation with World Foundation Modelscited by 80
- 2018Deep Generative Modeling of LiDAR Datacited by 70
- 2025GigaWorld-0: World Models as Data Engine to Empower Embodied AIcited by 60
- 2024LidarDM: Generative LiDAR Simulation in a Generated Worldcited by 53
- 2024RangeLDM: Fast Realistic LiDAR Point Cloud Generationcited by 45
- 2025ReconDreamer-RL: Enhancing Reinforcement Learning via Diffusion-based Scene Reconstructioncited by 27
- 2022Are We Ready for Vision-Centric Driving Streaming Perception? The ASAP Benchmarkcited by 24
- 2025UniFuture: A 4D Driving World Model for Future Generation and Perceptioncited by 21
- 2025Genesis: Multimodal Driving Scene Generation with Spatio-Temporal and Cross-Modal Consistencycited by 21
- 2025DriveGen3D: Boosting Feed-Forward Driving Scene Generation with Efficient Video Diffusioncited by 14
- 2025OmniGen: Unified Multimodal Sensor Generation for Autonomous Drivingcited by 5
Built on by
Citing papers per year: 2026: 9.
9 papers.
- 2026DriveDreamer-Policy: A Geometry-Grounded World-Action Model for Unified Generation and Planningcited by 17
- 2026GigaWorld-1: A Roadmap to Build World Models for Robot Policy Evaluationcited by 10
- 2026GigaWorld-Policy-0.5: A Faster and Stronger WAM Empowered by AutoResearchcited by 10
- 2026Is Forward Prediction Enough? Physical State Grounding for JEPA World Modelscited by 5
- 2026BilliardPhys-Bench: Benchmarking Physical Reasoning and Visual Dynamics of Multimodal LLMscited by 1
- 2026BadDreamer: Transferable Backdoor Attacks against Video World Models for Autonomous Drivingcited by 1
- 2026Multi-Agent Embodied Autonomous Driving (MAEAD): From V2X Information Exchange to Shared World Modelscited by 0
- 2026HelloWorld: Towards Practical Applications of Generative Driving World Modelscited by 0
- 2026CRISP: Fixing Flying Pixels in Latent LiDAR Generation via Diffusion Decodingcited by 0
- 2026CRISP: Fixing Flying Pixels in Latent LiDAR Generation via Diffusion Decoding8 Oct 2026
- 2026HelloWorld: Towards Practical Applications of Generative Driving World Models24 Sep 2026
- 2026Is Forward Prediction Enough? Physical State Grounding for JEPA World Models7 Aug 2026
- 2026GigaWorld-Policy-0.5: A Faster and Stronger WAM Empowered by AutoResearch15 Jul 2026
- 2026GigaWorld-1: A Roadmap to Build World Models for Robot Policy Evaluation2 Jul 2026
- 2026BadDreamer: Transferable Backdoor Attacks against Video World Models for Autonomous Driving19 Jun 2026
- 2026Multi-Agent Embodied Autonomous Driving (MAEAD): From V2X Information Exchange to Shared World Models11 Jun 2026
- 2026BilliardPhys-Bench: Benchmarking Physical Reasoning and Visual Dynamics of Multimodal LLMs29 May 2026
- 2026DriveDreamer-Policy: A Geometry-Grounded World-Action Model for Unified Generation and Planning2 Apr 2026
Similar, not cited
Close in wording, no citation either way
- 2024BEVWorld: A Multimodal World Simulator for Autonomous Driving via Scene-Level BEV Latents
- 2024X-Drive: Cross-modality consistent multi-sensor data synthesis for driving scenarios
- 2026UniDWM: Towards a Unified Driving World Model via Multifaceted Representation Learning
- 2026LMGenDrive: Bridging Multimodal Understanding and Generative World Modeling for End-to-End Driving
- 2025MoVieDrive: Urban Scene Synthesis with Multi-Modal Multi-View Video Diffusion Transformer
- 2026UniDrive-WM: Unified Understanding, Planning and Generation World Model for Autonomous Driving
- 2025MultiEditor: Controllable Multimodal Object Editing for Driving Scenarios Using 3D Gaussian Splatting Priors
- 2024HoloDrive: Holistic 2D-3D Multi-Modal Street Scene Generation for Autonomous Driving