mlsanityData to 7 Oct 2026

DreamForge: Motion-Aware Autoregressive Video Generation for Multi-View Driving Scenes

arXiv 2409.04003 ↗First posted 6 Sep 2024cs.CVcited by 34

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 1 more
Abstract (arXiv)

Recent advances in diffusion models have improved controllable streetscape generation and supported downstream perception and planning tasks. However, challenges remain in accurately modeling driving scenes and generating long videos. To alleviate these issues, we propose DreamForge, an advanced diffusion-based autoregressive video generation model tailored for 3D-controllable long-term generation. To enhance the lane and foreground generation, we introduce perspective guidance and integrate object-wise position encoding to incorporate local 3D correlation and improve foreground object modeling. We also propose motion-aware temporal attention to capture motion cues and appearance changes in videos. By leveraging motion frames and an autoregressive generation paradigm,we can autoregressively generate long videos (over 200 frames) using a model trained in short sequences, achieving superior quality compared to the baseline in 16-frame video evaluations. Finally, we integrate our method with the realistic simulator DriveArena to provide more reliable open-loop and closed-loop evaluations for vision-based driving agents. Project Page: https://pjlab-adg.github.io/DriveArena/dreamforge.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 70 of its 82 references that are in the map. First posted on arXiv 6 Sep 2024.

Builds on

70 of 82 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023SceneDM: Scene-level Multi-agent Trajectory Generation with Consistent Diffusion Models27 Nov
  2. 2023DrivingDiffusion: Layout-Guided multi-view driving scene video generation with latent diffusion model11 Oct
  3. 2023MagicDrive: Street View Generation with Diverse 3D Geometry Control4 Oct

Other references · 67

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2020Denoising Diffusion Implicit Modelscited by 9,655
  4. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  7. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
  8. 2022Scalable Diffusion Models with Transformerscited by 5,931
  9. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  10. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  11. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  12. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
38 more
  1. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  2. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  3. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,436
  4. 2022Video Diffusion Modelscited by 2,280
  5. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,198
  6. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  7. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,418
  8. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformerscited by 1,213
  9. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
  10. 2020Center-based 3D Object Detection and Trackingcited by 1,143
  11. 2022Planning-oriented Autonomous Drivingcited by 1,068
  12. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 618
  13. 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 613
  14. 2022PETR: Position Embedding Transformation for Multi-View 3D Object Detectioncited by 550
  15. 2021NuPlan: A closed-loop ML-based planning benchmark for autonomous vehiclescited by 485
  16. 2023UniPC: A Unified Predictor-Corrector Framework for Fast Sampling of Diffusion Modelscited by 441
  17. 2023DrivingGaussian: Composite Gaussian Splatting for Surrounding Dynamic Autonomous Driving Scenescited by 359
  18. 2024NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarkingcited by 343
  19. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 337
  20. 2022BEVFusion: A Simple and Robust LiDAR-Camera Fusion Frameworkcited by 334
  21. 2023Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Drivingcited by 309
  22. 2022MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolationcited by 307
  23. 2022Cross-view Transformers for real-time Map-view Semantic Segmentationcited by 264
  24. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Drivingcited by 258
  25. 2023Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving?cited by 242
  26. 2024StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Textcited by 242
  27. 2023SEINE: Short-to-Long Video Diffusion Model for Generative Transition and Predictioncited by 231
  28. 2023DiLu: A Knowledge-Driven Approach to Autonomous Driving with Large Language Modelscited by 204
  29. 2023A Survey on Video Diffusion Modelscited by 203
  30. 2023Panacea: Panoramic and Controllable Video Generation for Autonomous Drivingcited by 165
  31. 2023Drive Like a Human: Rethinking Autonomous Driving with Large Language Modelscited by 158
  32. 2023SUDS: Scalable Urban Dynamic Scenescited by 151
  33. 2024GenAD: Generalized Predictive Model for Autonomous Drivingcited by 150
  34. 2023Make Pixels Dance: High-Dynamic Video Generationcited by 140
  35. 2024ID-Animator: Zero-Shot Identity-Preserving Human Video Generationcited by 122
  36. 2023Street-View Image Generation from a Bird's-Eye View Layoutcited by 102
  37. 2023BEVControl: Accurately Controlling Street-view Elements with Multi-perspective Consistency via BEV Sketch Layoutcited by 102
  38. 2020DSGN: Deep Stereo Geometry Network for 3D Object Detectioncited by 100

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 19; 2026: 14.

34 papers.

Sort:

By area:

  1. 2025Controllable Video Generation: A Surveycited by 93
  2. 20253D and 4D World Modeling: A Surveycited by 64
  3. 2024UniMLVG: Unified Framework for Multi-view Long Video Generation with Comprehensive Control Capabilities for Autonomous Drivingcited by 27
  4. 20253D Scene Generation: A Surveycited by 24
  5. 2025WorldLens: Full-Spectrum Evaluations of Driving World Models in Real Worldcited by 22
  6. 2025X-Scene: Large-Scale Driving Scene Generation with High Fidelity and Flexible Controllabilitycited by 20
  7. 2025Simulating the Visual World with Artificial Intelligence: A Roadmapcited by 15
  8. 2025LiDARCrafter: Dynamic 4D World Modeling from LiDAR Sequencescited by 14
  9. 2025Generating Multimodal Driving Scenes via Next-Scene Predictioncited by 12
  10. 2025DiVE: Efficient Multi-View Driving Scenes Generation Based on Video Diffusion Transformercited by 10
  11. 2025STAGE: A Stream-Centric Generative World Model for Long-Horizon Driving-Scene Simulationcited by 5
  12. 2025RLGF: Reinforcement Learning with Geometric Feedback for Autonomous Driving Video Generationcited by 5
22 more
  1. 2025The Safety Challenge of World Models for Embodied AI Agents: A Reviewcited by 5
  2. 2026SPIRAL: Self-Evolving Action-Conditioned Video Generation via Reflective Planning Agentscited by 5
  3. 2026Is Your Driving World Model an All-Around Player?cited by 5
  4. 2025Vision-Centric 4D Occupancy Forecasting and Planning via Implicit Residual World Modelscited by 3
  5. 2026FAR-Drive: Frame-AutoRegressive Video Generation in Closed-Loop Autonomous Drivingcited by 3
  6. 2026HorizonDrive: Self-Corrective Autoregressive World Model for Long-horizon Driving Simulationcited by 3
  7. 2025AccidentSim: Generating Vehicle Collision Videos with Physically Realistic Collision Trajectories from Real-World Accident Reportscited by 2
  8. 2025Learning to Generate 4D LiDAR Sequencescited by 2
  9. 2026OmniDrive: An LLM-Choreographed Multi-Agent World Model with Unified Latent Co-Compression for Multi-View Driving Video Generationcited by 2
  10. 2025CVD-STORM: Cross-View Video Diffusion with Spatial-Temporal Reconstruction Model for Autonomous Drivingcited by 1
  11. 2026FrozenDrive: Zero-Shot Text-Guided Driving Scene Generation and Data Augmentation with Parameter-Free Frozen Diffusion Modelcited by 1
  12. 2026Point as Skeleton: Accumulated Point Cloud Enhanced Autoregressive Generation for Closed-Loop Autonomous Driving Simulationcited by 1
  13. 2025DriveCamSim: Generalizable Camera Simulation via Explicit Camera Modeling for Autonomous Drivingcited by 0
  14. 2025LiSTAR: Ray-Centric World Models for 4D LiDAR Sequences in Autonomous Drivingcited by 0
  15. 2025VLM as Strategist: Adaptive Generation of Safety-critical Testing Scenarios via Guided Diffusioncited by 0
  16. 2026LSA: Localized Semantic Alignment for Enhancing Temporal Consistency in Traffic Video Generationcited by 0
  17. 2026DriveFix: Spatio-Temporally Coherent Driving Scene Restorationcited by 0
  18. 2026OccDirector: Language-Guided Behavior and Interaction Generation in 4D Occupancy Spacecited by 0
  19. 2026OmniSCS: Omni Safety-Critical Scenario Synthesis for Autonomous Driving via a Fully Editable Driving Worldcited by 0
  20. 2026Quo Vadis, World Modeling?cited by 0
  21. 2026GeoFlow: Efficient Driving Video Generation via Geometry-Aligned Priorscited by 0
  22. 2026DreamStream: Towards Policy-Oriented Generative Simulation for End-to-End Drivingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024DreamDrive: Generative 4D Scene Modeling from Street View Images
  2. 2025WorldSplat: Gaussian-Centric Feed-Forward 4D Scene Generation for Autonomous Driving
  3. 2024DriveScape: Towards High-Resolution Controllable Multi-View Driving Video Generation
  4. 2025Epona: Autoregressive Diffusion World Model for Autonomous Driving
  5. 2025Glad: A Streaming Scene Generator for Autonomous Driving
  6. 2024MagicDrive-V2: High-Resolution Long Video Generation for Autonomous Driving with Adaptive Control
  7. 2025Rethinking Driving World Model as Synthetic Data Generator for Perception Tasks
  8. 2024DrivingWorld: Constructing World Model for Autonomous Driving via Video GPT