mlsanityData to 7 Oct 2026

MagicDrive3D: Controllable 3D Generation for Any-View Rendering in Street Scenes

arXiv 2405.14475 ↗First posted 23 May 2024cs.CV cs.AIcited by 70

On the map: World models → World models for autonomous driving

Abstract (arXiv)

Controllable generative models for images and videos have seen significant success, yet 3D scene generation, especially in unbounded scenarios like autonomous driving, remains underdeveloped. Existing methods lack flexible controllability and often rely on dense view data collection in controlled environments, limiting their generalizability across common datasets (e.g., nuScenes). In this paper, we introduce MagicDrive3D, a novel framework for controllable 3D street scene generation that combines video-based view synthesis with 3D representation (3DGS) generation. It supports multi-condition control, including road maps, 3D objects, and text descriptions. Unlike previous approaches that require 3D representation before training, MagicDrive3D first trains a multi-view video generation model to synthesize diverse street views. This method utilizes routinely collected autonomous driving data, reducing data acquisition challenges and enriching 3D scene generation. In the 3DGS generation step, we introduce Fault-Tolerant Gaussian Splatting to address minor errors and use monocular depth for better initialization, alongside appearance modeling to manage exposure discrepancies across viewpoints. Experiments show that MagicDrive3D generates diverse, high-quality 3D driving scenes, supports any-view rendering, and enhances downstream tasks like BEV segmentation, demonstrating its potential for autonomous driving simulation and beyond.

Key facts. Cited by 70 papers in the map (data through 7 Oct 2026). Placed in World models for autonomous driving, one of 11 directions in World models. Builds on 34 of its 56 references that are in the map. First posted on arXiv 23 May 2024.

Builds on

34 of 56 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving29 Nov
  2. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Driving18 Sep

Other references · 32

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  5. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
  6. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  7. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  8. 2021Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fieldscited by 2,013
  9. 2020NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collectionscited by 1,230
  10. 20234D Gaussian Splatting for Real-Time Dynamic Scene Renderingcited by 1,144
  11. 2023DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creationcited by 837
  12. 2023ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depthcited by 646
20 more
  1. 2022LION: Latent Point Diffusion Models for 3D Shape Generationcited by 552
  2. 2023Gaussian Splatting SLAMcited by 410
  3. 2023DrivingGaussian: Composite Gaussian Splatting for Surrounding Dynamic Autonomous Driving Scenescited by 359
  4. 2022Cross-view Transformers for real-time Map-view Semantic Segmentationcited by 264
  5. 2021Urban Radiance Fieldscited by 263
  6. 2023MagicDrive: Street View Generation with Diverse 3D Geometry Controlcited by 250
  7. 2023LucidDreamer: Domain-free Generation of 3D Gaussian Splatting Scenescited by 228
  8. 2024VastGaussian: Vast 3D Gaussians for Large Scene Reconstructioncited by 209
  9. 2023Panacea: Panoramic and Controllable Video Generation for Autonomous Drivingcited by 165
  10. 2023WonderJourney: Going from Anywhere to Everywherecited by 157
  11. 2022GAUDI: A Neural Architect for Immersive 3D Scene Generationcited by 136
  12. 2021Unconstrained Scene Generation with Locally Conditioned Radiance Fieldscited by 130
  13. 2023S-NeRF: Neural Radiance Fields for Street Viewscited by 116
  14. 2021Geometry-Free View Synthesis: Transformers and no 3D Priorscited by 99
  15. 2023NeuralField-LDM: Scene Generation with Hierarchical Latent Diffusion Modelscited by 76
  16. 2023GeoDiffusion: Text-Prompted Geometric Control for Object Detection Data Generationcited by 56
  17. 2023DynPoint: Dynamic Neural Point For View Synthesiscited by 37
  18. 2024DetDiffusion: Synergizing Generative and Perceptive Models for Enhanced Data Generation and Perceptioncited by 32
  19. 2023Pseudo-Generalized Dynamic View Synthesis from a Videocited by 27
  20. 2023Towards Viewpoint Robustness in Bird's Eye View Segmentationcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 13; 2025: 40; 2026: 17.

70 papers.

Sort:

By area:

  1. 2024DriveDreamer4D: World Models Are Effective Data Machines for 4D Driving Scene Representationcited by 123
  2. 2024MagicDrive-V2: High-Resolution Long Video Generation for Autonomous Driving with Adaptive Controlcited by 90
  3. 2024ReconDreamer: Crafting World Models for Driving Scene Reconstruction via Online Restorationcited by 89
  4. 2025Cosmos-Drive-Dreams: Scalable Synthetic Driving Data Generation with World Foundation Modelscited by 80
  5. 2025A Comprehensive Survey on World Models for Embodied AIcited by 65
  6. 20253D and 4D World Modeling: A Surveycited by 64
  7. 2025A Survey of World Models for Autonomous Drivingcited by 60
  8. 2024InfiniCube: Unbounded and Controllable Dynamic 3D Driving Scene Generation with World-Guided Video Modelscited by 53
  9. 2025A Survey: Learning Embodied Intelligence from Physical Simulators and World Modelscited by 50
  10. 2024StreetCrafter: Street View Synthesis with Controllable Video Diffusion Modelscited by 49
  11. 2024EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotionscited by 44
  12. 2024Efficient Diffusion Models: A Comprehensive Survey from Principles to Practicescited by 39
38 more
  1. 2025DiST-4D: Disentangled Spatiotemporal Diffusion with Metric Depth for 4D Driving Scene Generationcited by 39
  2. 2025Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysiscited by 37
  3. 2024DreamDrive: Generative 4D Scene Modeling from Street View Imagescited by 35
  4. 2024DreamForge: Motion-Aware Autoregressive Video Generation for Multi-View Driving Scenescited by 34
  5. 2024Omni-Scene: Omni-Gaussian Representation for Ego-Centric Sparse-View Scene Reconstructioncited by 32
  6. 2025Detect Anything 3D in the Wildcited by 32
  7. 2025The Role of World Models in Shaping Autonomous Driving: A Comprehensive Surveycited by 29
  8. 2025Genesis: Multimodal Driving Scene Generation with Spatio-Temporal and Cross-Modal Consistencycited by 21
  9. 2025DriveX: Omni Scene Modeling for Learning Generalizable World Knowledge in Autonomous Drivingcited by 20
  10. 2025X-Scene: Large-Scale Driving Scene Generation with High Fidelity and Flexible Controllabilitycited by 20
  11. 2024Stag-1: Towards Realistic 4D Driving Simulation with Video Generation Modelcited by 18
  12. 2025MiLA: Multi-view Intensive-fidelity Long-term Video Generation World Model for Autonomous Drivingcited by 18
  13. 2025WorldGen: From Text to Traversable and Interactive 3D Worldscited by 18
  14. 2025Challenger: Affordable Adversarial Driving Video Generationcited by 16
  15. 2025ORV: 4D Occupancy-centric Robot Video Generationcited by 16
  16. 2025CoGen: 3D Consistent Video Generation via Adaptive Conditioning for Autonomous Drivingcited by 12
  17. 2025WorldSplat: Gaussian-Centric Feed-Forward 4D Scene Generation for Autonomous Drivingcited by 11
  18. 2024Seeing Beyond Views: Multi-View Driving Scene Video Generation with Holistic Attentioncited by 10
  19. 2025Learning-based 3D Reconstruction in Autonomous Driving: A Comprehensive Surveycited by 9
  20. 2025MuDG: Taming Multi-modal Diffusion with Gaussian Splatting for Urban Scene Reconstructioncited by 8
  21. 20254D Driving Scene Generation With Stereo Forcingcited by 8
  22. 2024Unified Triplet-Level Hallucination Evaluation for Large Vision-Language Modelscited by 7
  23. 2025Generative AI for Autonomous Driving: Frontiers and Opportunitiescited by 7
  24. 2025AirScape: An Aerial Generative World Model with Motion Controllabilitycited by 7
  25. 2025ECCV 2024 W-CODA: 1st Workshop on Multimodal Perception and Comprehension of Corner Cases in Autonomous Drivingcited by 5
  26. 2025LSD-3D: Large-Scale 3D Driving Scene Generation with Geometry Groundingcited by 5
  27. 2025VR-Drive: Viewpoint-Robust End-to-End Driving with Feed-Forward 3D Gaussian Splattingcited by 5
  28. 2025Spatial Retrieval Augmented Autonomous Drivingcited by 5
  29. 2025OmniGen: Unified Multimodal Sensor Generation for Autonomous Drivingcited by 5
  30. 2025MonoPlace3D: Learning 3D-Aware Object Placement for 3D Monocular Detectioncited by 4
  31. 2026RAYNOVA: Scale-Temporal Autoregressive World Modeling in Ray Spacecited by 4
  32. 2026HorizonForge: Driving Scene Editing with Any Trajectories and Any Vehiclescited by 3
  33. 2026Motion Forcing: A Decoupled Framework for Robust Video Generation in Motion Dynamicscited by 3
  34. 2026FAR-Drive: Frame-AutoRegressive Video Generation in Closed-Loop Autonomous Drivingcited by 3
  35. 2026HorizonDrive: Self-Corrective Autoregressive World Model for Long-horizon Driving Simulationcited by 3
  36. 2025Gaussian Splatting is an Effective Data Generator for 3D Object Detectioncited by 2
  37. 2025RGE-GS: Reward-Guided Expansive Driving Scene Reconstruction via Diffusion Priorscited by 2
  38. 2025AutoScape: Geometry-Consistent Long-Horizon Scene Generationcited by 2

50 most cited of 70 All 70 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025DriveGen3D: Boosting Feed-Forward Driving Scene Generation with Efficient Video Diffusion
  2. 2024PerLDiff: Controllable Street View Synthesis Using Perspective-Layout Diffusion Models
  3. 20263DCarGen: Scalable 3D Car Generation via 3D-consistent Multi-view Synthesis
  4. 20253D Scene Generation: A Survey
  5. 2024HoloDrive: Holistic 2D-3D Multi-Modal Street Scene Generation for Autonomous Driving
  6. 2025SimWorld: A Unified Benchmark for Simulator-Conditioned Scene Generation via World Model
  7. 2024DriveScape: Towards High-Resolution Controllable Multi-View Driving Video Generation
  8. 2025MeSS: City Mesh-Guided Outdoor Scene Generation with Cross-View Consistent Diffusion