mlsanityData to 7 Oct 2026

Unleashing Generalization of End-to-End Autonomous Driving with Controllable Long Video Generation

arXiv 2406.01349 ↗First posted 3 Jun 2024cs.CVcited by 54

On the map: World models → World models for autonomous driving

also in 1 more
Abstract (arXiv)

Using generative models to synthesize new data has become a de-facto standard in autonomous driving to address the data scarcity issue. Though existing approaches are able to boost perception models, we discover that these approaches fail to improve the performance of planning of end-to-end autonomous driving models as the generated videos are usually less than 8 frames and the spatial and temporal inconsistencies are not negligible. To this end, we propose Delphi, a novel diffusion-based long video generation method with a shared noise modeling mechanism across the multi-views to increase spatial consistency, and a feature-aligned module to achieves both precise controllability and temporal consistency. Our method can generate up to 40 frames of video without loss of consistency which is about 5 times longer compared with state-of-the-art methods. Instead of randomly generating new data, we further design a sampling policy to let Delphi generate new data that are similar to those failure cases to improve the sample efficiency. This is achieved by building a failure-case driven framework with the help of pre-trained visual language models. Our extensive experiment demonstrates that our Delphi generates a higher quality of long videos surpassing previous state-of-the-art methods. Consequentially, with only generating 4% of the training dataset size, our framework is able to go beyond perception and prediction tasks, for the first time to the best of our knowledge, boost the planning performance of the end-to-end autonomous driving model by a margin of 25%.

Key facts. Cited by 54 papers in the map (data through 7 Oct 2026). Placed in World models for autonomous driving, one of 11 directions in World models. Builds on 44 of its 48 references that are in the map. First posted on arXiv 3 Jun 2024.

Builds on

44 of 48 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generation11 Mar
  2. 2023Visual Point Cloud Forecasting enables Scalable Autonomous Driving29 Dec
  3. 2023Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving29 Nov
  4. 2023ADriver-I: A General World Model for Autonomous Driving22 Nov
  5. 2023GAIA-1: A Generative World Model for Autonomous Driving29 Sep
  6. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Driving18 Sep

Other references · 38

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2020Denoising Diffusion Implicit Modelscited by 9,655
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  7. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  8. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  9. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  10. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  11. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  12. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
26 more
  1. 2022Planning-oriented Autonomous Drivingcited by 1,068
  2. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 958
  3. 2022Pseudo Numerical Methods for Diffusion Models on Manifoldscited by 695
  4. 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 613
  5. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 523
  6. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Modelscited by 463
  7. 2016Generating Videos with Scene Dynamicscited by 376
  8. 2021Multi-Modal Fusion Transformer for End-to-End Autonomous Drivingcited by 372
  9. 2022ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learningcited by 365
  10. 2023VideoFusion: Decomposed Diffusion Models for High-Quality Video Generationcited by 330
  11. 2023Exploring Object-Centric Temporal Modeling for Efficient Multi-View 3D Object Detectioncited by 313
  12. 2024Long-CLIP: Unlocking the Long-Text Capability of CLIPcited by 298
  13. 2015Deep multi-scale video prediction beyond mean square errorcited by 297
  14. 2022Flexible Diffusion Modeling of Long Videoscited by 294
  15. 2024VADv2: End-to-End Vectorized Autonomous Driving via Probabilistic Planningcited by 287
  16. 2023MagicDrive: Street View Generation with Diverse 3D Geometry Controlcited by 250
  17. 2018Stochastic Video Generation with a Learned Priorcited by 244
  18. 2016Temporal Generative Adversarial Nets with Singular Value Clippingcited by 198
  19. 2023Rethinking the Open-Loop Evaluation of End-to-End Autonomous Driving in nuScenescited by 191
  20. 2023FreeNoise: Tuning-Free Longer Video Diffusion via Noise Reschedulingcited by 173
  21. 2023Panacea: Panoramic and Controllable Video Generation for Autonomous Drivingcited by 165
  22. 2022Diffusion Models for Video Prediction and Infillingcited by 134
  23. 2024ConsistI2V: Enhancing Visual Consistency for Image-to-Video Generationcited by 130
  24. 2023Street-View Image Generation from a Bird's-Eye View Layoutcited by 102
  25. 2023BEVControl: Accurately Controlling Street-view Elements with Multi-perspective Consistency via BEV Sketch Layoutcited by 102
  26. 2024AIDE: An Automatic Data Engine for Object Detection in Autonomous Drivingcited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 34; 2026: 9.

54 papers.

Sort:

By area:

  1. 2025GAIA-2: A Controllable Multi-View Generative World Model for Autonomous Drivingcited by 178
  2. 2025Pseudo-Simulation for Autonomous Drivingcited by 141
  3. 2025Controllable Video Generation: A Surveycited by 93
  4. 2024MagicDrive-V2: High-Resolution Long Video Generation for Autonomous Driving with Adaptive Controlcited by 90
  5. 2025Cosmos-Drive-Dreams: Scalable Synthetic Driving Data Generation with World Foundation Modelscited by 80
  6. 2024Driving in the Occupancy World: Vision-Centric 4D Occupancy Forecasting and Planning via World Models for Autonomous Drivingcited by 67
  7. 2025A Comprehensive Survey on World Models for Embodied AIcited by 65
  8. 20253D and 4D World Modeling: A Surveycited by 64
  9. 2025A Survey of World Models for Autonomous Drivingcited by 60
  10. 2025HERMES: A Unified Self-Driving World Model for Simultaneous 3D Scene Understanding and Generationcited by 56
  11. 2024Doe-1: Closed-Loop Autonomous Driving with Large World Modelcited by 51
  12. 2025DriveLaW:Unifying Planning and Video Generation in a Latent Driving Worldcited by 43
38 more
  1. 2024DreamForge: Motion-Aware Autoregressive Video Generation for Multi-View Driving Scenescited by 34
  2. 2024DrivingSphere: Building a High-fidelity 4D World for Closed-loop Simulationcited by 34
  3. 2024DiVE: DiT-based Video Generation with Enhanced Controlcited by 31
  4. 2024InfinityDrive: Breaking Time Limits in Driving World Modelscited by 21
  5. 2025Genesis: Multimodal Driving Scene Generation with Spatio-Temporal and Cross-Modal Consistencycited by 21
  6. 2025X-Scene: Large-Scale Driving Scene Generation with High Fidelity and Flexible Controllabilitycited by 20
  7. 2025GeoDrive: 3D Geometry-Informed Driving World Model with Precise Action Controlcited by 19
  8. 2025Reinforced Refinement with Self-Aware Expansion for End-to-End Autonomous Drivingcited by 19
  9. 2026Flow caching for autoregressive video generationcited by 19
  10. 2025MiLA: Multi-view Intensive-fidelity Long-term Video Generation World Model for Autonomous Drivingcited by 18
  11. 2024DriveEditor: A Unified 3D Information-Guided Framework for Controllable Object Editing in Driving Scenescited by 17
  12. 2025Simulating the Visual World with Artificial Intelligence: A Roadmapcited by 15
  13. 2024Exploring the Interplay Between Video Generation and World Models in Autonomous Driving: A Surveycited by 14
  14. 2024Bench2Drive-R: Turning Real World Data into Reactive Closed-Loop Autonomous Driving Benchmark by Generative Modelcited by 14
  15. 2025Rethinking Driving World Model as Synthetic Data Generator for Perception Taskscited by 14
  16. 2025Unraveling the Effects of Synthetic Data on End-to-End Autonomous Drivingcited by 12
  17. 2025CoGen: 3D Consistent Video Generation via Adaptive Conditioning for Autonomous Drivingcited by 12
  18. 2025Decoupled Diffusion Sparks Adaptive Scene Generationcited by 12
  19. 2025EMMA: Generalizing Real-World Robot Manipulation via Generative Visual Transfercited by 12
  20. 2025WorldSplat: Gaussian-Centric Feed-Forward 4D Scene Generation for Autonomous Drivingcited by 11
  21. 2025Learning-based 3D Reconstruction in Autonomous Driving: A Comprehensive Surveycited by 9
  22. 2025SceneCrafter: Controllable Multi-View Driving Scene Editingcited by 9
  23. 2026UniDriveDreamer: A Single-Stage Multimodal World Model for Autonomous Drivingcited by 9
  24. 2025Generative AI for Autonomous Driving: Frontiers and Opportunitiescited by 7
  25. 2025DualDiff+: Dual-Branch Diffusion for High-Fidelity Video Generation with Reward Guidancecited by 6
  26. 2025DualDiff: Dual-branch Diffusion Model for Autonomous Driving with Semantic Fusioncited by 6
  27. 2025RLGF: Reinforcement Learning with Geometric Feedback for Autonomous Driving Video Generationcited by 5
  28. 2025Spatial Retrieval Augmented Autonomous Drivingcited by 5
  29. 2025OmniGen: Unified Multimodal Sensor Generation for Autonomous Drivingcited by 5
  30. 2026Xiaomi Auto World Model: A Joint World Model Integrating Reconstruction and Generation for Autonomous Drivingcited by 5
  31. 2025Drive&Gen: Co-Evaluating End-to-End Driving and Video Generation Modelscited by 4
  32. 2026ProDrive: Proactive Planning for Autonomous Driving via Ego-Environment Co-Evolutioncited by 4
  33. 2025NoiseController: Towards Consistent Multi-view Video Generation via Noise Decomposition and Collaborationcited by 3
  34. 2025Generative AI for Autonomous Driving: A Reviewcited by 3
  35. 2026HERMES++: Toward a Unified Driving World Model for 3D Scene Understanding and Generationcited by 3
  36. 2025CorrectAD: A Self-Correcting Agentic System to Improve End-to-end Planning in Autonomous Drivingcited by 2
  37. 2025Optimization-Guided Diffusion for Interactive Scene Generationcited by 2
  38. 2026OmniDrive: An LLM-Choreographed Multi-Agent World Model with Unified Latent Co-Compression for Multi-View Driving Video Generationcited by 2

50 most cited of 54 All 54 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026FAR-Drive: Frame-AutoRegressive Video Generation in Closed-Loop Autonomous Driving
  2. 2024GenDDS: Generating Diverse Driving Video Scenarios with Prompt-to-Video Generative Model
  3. 2025STAGE: A Stream-Centric Generative World Model for Long-Horizon Driving-Scene Simulation
  4. 2026DrivingGen: A Comprehensive Benchmark for Generative Video World Models in Autonomous Driving
  5. 2024MyGo: Consistent and Controllable Multi-View Driving Video Generation with Camera Control
  6. 2023DrivingDiffusion: Layout-Guided multi-view driving scene video generation with latent diffusion model
  7. 2024DreamDrive: Generative 4D Scene Modeling from Street View Images
  8. 2024DriveGenVLM: Real-world Video Generation for Vision Language Model based Autonomous Driving