mlsanityData to 14 Oct 2026

DrivingDiffusion: Layout-Guided multi-view driving scene video generation with latent diffusion model

arXiv 2310.07771 ↗First posted 11 Oct 2023cs.CV cs.AIcited by 75

On the map: Diffusion models → Diffusion and flow policies for robotics

Abstract (arXiv)

With the increasing popularity of autonomous driving based on the powerful and unified bird's-eye-view (BEV) representation, a demand for high-quality and large-scale multi-view video data with accurate annotation is urgently required. However, such large-scale multi-view data is hard to obtain due to expensive collection and annotation costs. To alleviate the problem, we propose a spatial-temporal consistent diffusion framework DrivingDiffusion, to generate realistic multi-view videos controlled by 3D layout. There are three challenges when synthesizing multi-view videos given a 3D layout: How to keep 1) cross-view consistency and 2) cross-frame consistency? 3) How to guarantee the quality of the generated instances? Our DrivingDiffusion solves the problem by cascading the multi-view single-frame image generation step, the single-view video generation step shared by multiple cameras, and post-processing that can handle long video generation. In the multi-view model, the consistency of multi-view images is ensured by information exchange between adjacent cameras. In the temporal model, we mainly query the information that needs attention in subsequent frame generation from the multi-view images of the first frame. We also introduce the local prompt to effectively improve the quality of generated instances. In post-processing, we further enhance the cross-view consistency of subsequent frames and extend the video length by employing temporal sliding window algorithm. Without any extra cost, our model can generate large-scale realistic multi-camera driving videos in complex urban scenes, fueling the downstream driving tasks. The code will be made publicly available.

Key facts. Cited by 75 papers in the map (data through 14 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 42 of its 45 references that are in the map. First posted on arXiv 11 Oct 2023.

Builds on

42 of 45 references.

References · 42

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2020Denoising Diffusion Implicit Modelscited by 9,781
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  7. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  8. 2017Neural Discrete Representation Learningcited by 4,712
  9. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  10. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  11. 2016Improved Techniques for Training GANscited by 2,970
  12. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
30 more
  1. 2018PointPillars: Fast Encoders for Object Detection from Point Cloudscited by 1,715
  2. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,476
  3. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,299
  4. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  5. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,234
  6. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformerscited by 1,218
  7. 2020Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3Dcited by 1,044
  8. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 985
  9. 2022BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representationcited by 850
  10. 2021Vector-quantized Image Modeling with Improved VQGANcited by 658
  11. 2022PETR: Position Embedding Transformation for Multi-View 3D Object Detectioncited by 552
  12. 2022CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformerscited by 307
  13. 2022Cross-view Transformers for real-time Map-view Semantic Segmentationcited by 264
  14. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Drivingcited by 261
  15. 2023LayoutDiffusion: Controllable Diffusion Model for Layout-to-image Generationcited by 259
  16. 2021GODIVA: Generating Open-DomaIn Videos from nAtural Descriptionscited by 256
  17. 2021N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtioncited by 253
  18. 2021FIERY: Future Instance Prediction in Bird's-Eye View from Surround Monocular Camerascited by 239
  19. 2020PnPNet: End-to-End Perception and Prediction with Tracking in the Loopcited by 144
  20. 2022ViP3D: End-to-end Visual Trajectory Prediction via 3D Agent Queriescited by 117
  21. 2023Street-View Image Generation from a Bird's-Eye View Layoutcited by 102
  22. 2022MUTR3D: A Multi-camera Tracking Framework via 3D-to-2D Queriescited by 91
  23. 2021Monocular Quasi-Dense 3D Object Trackingcited by 81
  24. 2023LayoutDiffuse: Adapting Foundational Diffusion Models for Layout-to-Image Generationcited by 65
  25. 2020Object-Centric Image Generation from Layoutscited by 57
  26. 2021Context-Aware Layout to Image Generation with Enhanced Object Appearancecited by 41
  27. 2023CAPE: Camera View Position Embedding for Multi-View 3D Object Detectioncited by 40
  28. 2023TBP-Former: Learning Temporal Bird's-Eye-View Pyramid for Joint Perception and Prediction in Vision-Centric Autonomous Drivingcited by 30
  29. 2022Perceive, Interact, Predict: Learning Dynamic and Static Clues for End-to-End Motion Predictioncited by 28
  30. 2023ByteTrackV2: 2D and 3D Multi-Object Tracking by Associating Every Detection Boxcited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 10; 2024: 38; 2025: 22; 2026: 5.

75 papers.

Sort:

By area:

  1. 2023OccWorld: Learning a 3D Occupancy World Model for Autonomous Drivingcited by 233
  2. 2024DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generationcited by 191
  3. 2024GenAD: Generalized Predictive Model for Autonomous Drivingcited by 150
  4. 2023LLM4Drive: A Survey of Large Language Models for Autonomous Drivingcited by 136
  5. 2023Vision Language Models in Autonomous Driving: A Survey and Outlookcited by 126
  6. 2024Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyondcited by 111
  7. 2023Visual Point Cloud Forecasting enables Scalable Autonomous Drivingcited by 99
  8. 2024DriveArena: A Closed-loop Generative Simulation Platform for Autonomous Drivingcited by 98
  9. 2023WoVoGen: World Volume-aware Diffusion for Controllable Multi-camera Driving Scene Generationcited by 93
  10. 2024Editable Scene Simulation for Autonomous Driving via Collaborative LLM-Agentscited by 91
  11. 2024Pandora: Towards General World Model with Natural Language Actions and Video Statescited by 86
  12. 2024OccLLaMA: An Occupancy-Language-Action Generative World Model for Autonomous Drivingcited by 83
38 more
  1. 2024Driving in the Occupancy World: Vision-Centric 4D Occupancy Forecasting and Planning via World Models for Autonomous Drivingcited by 69
  2. 2024EchoScene: Indoor Scene Generation via Information Echo over Scene Graph Diffusioncited by 67
  3. 20253D and 4D World Modeling: A Surveycited by 67
  4. 2024SCube: Instant Large-Scale Scene Reconstruction using VoxSplatscited by 60
  5. 2025A Survey of World Models for Autonomous Drivingcited by 60
  6. 2024Doe-1: Closed-Loop Autonomous Driving with Large World Modelcited by 51
  7. 2024DiP-GO: A Diffusion Pruner via Few-step Gradient Optimizationcited by 43
  8. 2024Efficient Diffusion Models: A Comprehensive Survey from Principles to Practicescited by 41
  9. 2024Solving Motion Planning Tasks with a Scalable Generative Modelcited by 40
  10. 2024DreamForge: Motion-Aware Autoregressive Video Generation for Multi-View Driving Scenescited by 35
  11. 2023Empowering Autonomous Driving with Large Language Models: A Safety Perspectivecited by 34
  12. 2024SimGen: Simulator-conditioned Driving Scene Generationcited by 34
  13. 2024SubjectDrive: Scaling Generative Data in Autonomous Driving via Subject Controlcited by 32
  14. 2025Vid2Sim: Realistic and Interactive Simulation from Video for Urban Navigationcited by 30
  15. 2023Towards Knowledge-driven Autonomous Drivingcited by 29
  16. 2024AutoVFX: Physically Realistic Video Editing from Natural Language Instructionscited by 27
  17. 2024Forging Vision Foundation Models for Autonomous Driving: Challenges, Methodologies, and Opportunitiescited by 24
  18. 2024DriveScape: Towards High-Resolution Controllable Multi-View Driving Video Generationcited by 22
  19. 2025VR-Robo: A Real-to-Sim-to-Real Framework for Visual Robot Navigation and Locomotioncited by 22
  20. 2024BEVWorld: A Multimodal World Simulator for Autonomous Driving via Scene-Level BEV Latentscited by 21
  21. 2024InfinityDrive: Breaking Time Limits in Driving World Modelscited by 21
  22. 2025GeoDrive: 3D Geometry-Informed Driving World Model with Precise Action Controlcited by 19
  23. 2024GGS: Generalizable Gaussian Splatting for Lane Switching in Autonomous Drivingcited by 17
  24. 2024DriveEditor: A Unified 3D Information-Guided Framework for Controllable Object Editing in Driving Scenescited by 17
  25. 2023Applications of Large Scale Foundation Models for Autonomous Drivingcited by 14
  26. 2024SyntheOcc: Synthesize Geometric-Controlled Street View Images through 3D Semantic MPIscited by 14
  27. 2024Exploring the Interplay Between Video Generation and World Models in Autonomous Driving: A Surveycited by 14
  28. 2024Bench2Drive-R: Turning Real World Data into Reactive Closed-Loop Autonomous Driving Benchmark by Generative Modelcited by 14
  29. 2024Physical Informed Driving World Modelcited by 13
  30. 2025VideoPanda: Video Panoramic Diffusion with Multi-view Attentioncited by 13
  31. 2024PerLDiff: Controllable Street View Synthesis Using Perspective-Layout Diffusion Modelscited by 12
  32. 2024Driving View Synthesis on Free-form Trajectories with Generative Priorcited by 11
  33. 2025ProphetDWM: A Driving World Model for Rolling Out Future Actions and Videoscited by 11
  34. 2023Layered Rendering Diffusion Model for Controllable Zero-Shot Image Synthesiscited by 9
  35. 2024Slight Corruption in Pre-training Data Makes Better Diffusion Modelscited by 9
  36. 2024World Models: The Safety Perspectivecited by 9
  37. 2025SceneCrafter: Controllable Multi-View Driving Scene Editingcited by 9
  38. 2023Prospective Role of Foundation Models in Advancing Autonomous Vehiclescited by 7

50 most cited of 75 All 75 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025DiVE: Efficient Multi-View Driving Scenes Generation Based on Video Diffusion Transformer
  2. 2025DriveGen3D: Boosting Feed-Forward Driving Scene Generation with Efficient Video Diffusion
  3. 2025MoVieDrive: Urban Scene Synthesis with Multi-Modal Multi-View Video Diffusion Transformer
  4. 2025WorldSplat: Gaussian-Centric Feed-Forward 4D Scene Generation for Autonomous Driving
  5. 2025X-Scene: Large-Scale Driving Scene Generation with High Fidelity and Flexible Controllability
  6. 2024DreamDrive: Generative 4D Scene Modeling from Street View Images
  7. 2026SEM-ROVER: Semantic Voxel-Guided Diffusion for Large-Scale Driving Scene Generation
  8. 2024Unleashing Generalization of End-to-End Autonomous Driving with Controllable Long Video Generation