mlsanityData to 7 Oct 2026

MagicDrive: Street View Generation with Diverse 3D Geometry Control

arXiv 2310.02601 ↗First posted 4 Oct 2023cs.CV cs.AIcited by 250

On the map: Diffusion models → Diffusion and flow policies for robotics

Abstract (arXiv)

Recent advancements in diffusion models have significantly enhanced the data synthesis with 2D control. Yet, precise 3D control in street view generation, crucial for 3D perception tasks, remains elusive. Specifically, utilizing Bird's-Eye View (BEV) as the primary condition often leads to challenges in geometry control (e.g., height), affecting the representation of object shapes, occlusion patterns, and road surface elevations, all of which are essential to perception data synthesis, especially for 3D object detection tasks. In this paper, we introduce MagicDrive, a novel street view generation framework, offering diverse 3D geometry controls including camera poses, road maps, and 3D bounding boxes, together with textual descriptions, achieved through tailored encoding strategies. Besides, our design incorporates a cross-view attention module, ensuring consistency across multiple camera views. With MagicDrive, we achieve high-fidelity street-view image & video synthesis that captures nuanced 3D geometry and various scene descriptions, enhancing tasks like BEV segmentation and 3D object detection.

Key facts. Cited by 250 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 40 of its 54 references that are in the map. First posted on arXiv 4 Oct 2023.

Builds on

40 of 54 references.

References · 40

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  7. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  8. 2022Classifier-Free Diffusion Guidancecited by 5,717
  9. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  10. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  11. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  12. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
28 more
  1. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  2. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  3. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 958
  4. 2022BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representationcited by 844
  5. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 774
  6. 2021BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-Viewcited by 711
  7. 2022Compositional Visual Generation with Composable Diffusion Modelscited by 619
  8. 2023UniPC: A Unified Predictor-Corrector Framework for Fast Sampling of Diffusion Modelscited by 441
  9. 2022DiffusionDet: Diffusion Model for Object Detectioncited by 354
  10. 2023Composer: Creative and Controllable Image Synthesis with Composable Conditionscited by 303
  11. 2022Cross-view Transformers for real-time Map-view Semantic Segmentationcited by 264
  12. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Drivingcited by 258
  13. 2022Imagen Editor and EditBench: Advancing and Evaluating Text-Guided Image Inpaintingcited by 229
  14. 2023MVDiffusion: Enabling Holistic Multi-view Image Generation with Correspondence-Aware Diffusioncited by 210
  15. 2023HIVE: Harnessing Human Feedback for Instructional Visual Editingcited by 172
  16. 2023DDP: Diffusion Model for Dense Visual Predictioncited by 148
  17. 2022Semantic Image Synthesis via Diffusion Modelscited by 131
  18. 2023DatasetDM: Synthesizing Data with Perception Annotations Using Diffusion Modelscited by 122
  19. 2023Consistent View Synthesis with Pose-Guided Diffusion Modelscited by 117
  20. 2023Street-View Image Generation from a Bird's-Eye View Layoutcited by 102
  21. 2023BEVControl: Accurately Controlling Street-view Elements with Multi-perspective Consistency via BEV Sketch Layoutcited by 102
  22. 2022CODA: A Real-World Road Corner Case Dataset for Object Detection in Autonomous Drivingcited by 100
  23. 2021MultiSiam: Self-supervised Multi-instance Siamese Representation Learning for Autonomous Drivingcited by 41
  24. 2023Mixed Autoencoder for Self-supervised Visual Representation Learningcited by 38
  25. 2023DiffGuard: Semantic Mismatch-Guided Out-of-Distribution Detection using Pre-trained Diffusion Modelscited by 25
  26. 2022Are We Ready for Vision-Centric Driving Streaming Perception? The ASAP Benchmarkcited by 24
  27. 2022Task-Customized Self-Supervised Pre-training with Scalable Dynamic Routingcited by 18
  28. 2023MetaBEV: Solving Sensor Failures for BEV Detection and Map Segmentationcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 13; 2024: 60; 2025: 108; 2026: 69.

250 papers.

Sort:

By area:

  1. 2023Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Drivingcited by 309
  2. 2023OccWorld: Learning a 3D Occupancy World Model for Autonomous Drivingcited by 230
  3. 2024DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generationcited by 186
  4. 2024GenAD: Generalized Predictive Model for Autonomous Drivingcited by 150
  5. 2023LLM4Drive: A Survey of Large Language Models for Autonomous Drivingcited by 136
  6. 2025Epona: Autoregressive Diffusion World Model for Autonomous Drivingcited by 115
  7. 2024Eyes Closed, Safety On: Protecting Multimodal LLMs via Image-to-Text Transformationcited by 114
  8. 2023Mixture of Cluster-conditional LoRA Experts for Vision-language Instruction Tuningcited by 106
  9. 2024UniScene: Unified Occupancy-centric Driving Scene Generationcited by 102
  10. 2024DriveArena: A Closed-loop Generative Simulation Platform for Autonomous Drivingcited by 98
  11. 2023WoVoGen: World Volume-aware Diffusion for Controllable Multi-camera Driving Scene Generationcited by 93
  12. 2025Controllable Video Generation: A Surveycited by 93
38 more
  1. 2024Editable Scene Simulation for Autonomous Driving via Collaborative LLM-Agentscited by 91
  2. 2025HunyuanWorld 1.0: Generating Immersive, Explorable, and Interactive 3D Worlds from Words or Pixelscited by 91
  3. 2024MagicDrive-V2: High-Resolution Long Video Generation for Autonomous Driving with Adaptive Controlcited by 90
  4. 2025Cosmos-Drive-Dreams: Scalable Synthetic Driving Data Generation with World Foundation Modelscited by 80
  5. 2024OccSora: 4D Occupancy Generation Models as World Simulators for Autonomous Drivingcited by 76
  6. 2024Automated Evaluation of Large Vision-Language Models on Self-driving Corner Casescited by 75
  7. 2024MagicDrive3D: Controllable 3D Generation for Any-View Rendering in Street Scenescited by 70
  8. 2024GaussianWorld: Gaussian World Model for Streaming 3D Occupancy Predictioncited by 66
  9. 20253D and 4D World Modeling: A Surveycited by 64
  10. 2025A Survey of World Models for Autonomous Drivingcited by 60
  11. 2025GigaWorld-0: World Models as Data Engine to Empower Embodied AIcited by 60
  12. 2025HERMES: A Unified Self-Driving World Model for Simultaneous 3D Scene Understanding and Generationcited by 56
  13. 2024Unleashing Generalization of End-to-End Autonomous Driving with Controllable Long Video Generationcited by 54
  14. 2024Doe-1: Closed-Loop Autonomous Driving with Large World Modelcited by 51
  15. 2024StreetCrafter: Street View Synthesis with Controllable Video Diffusion Modelscited by 49
  16. 2024EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotionscited by 44
  17. 2024DiP-GO: A Diffusion Pruner via Few-step Gradient Optimizationcited by 43
  18. 2025GigaBrain-0: A World Model-Powered Vision-Language-Action Modelcited by 43
  19. 2025DriveLaW:Unifying Planning and Video Generation in a Latent Driving Worldcited by 43
  20. 2024Streetscapes: Large-scale Consistent Street View Generation Using Autoregressive Video Diffusioncited by 39
  21. 2024Efficient Diffusion Models: A Comprehensive Survey from Principles to Practicescited by 39
  22. 2025DiST-4D: Disentangled Spatiotemporal Diffusion with Metric Depth for 4D Driving Scene Generationcited by 39
  23. 2025Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysiscited by 37
  24. 2026DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generationcited by 36
  25. 2024DreamDrive: Generative 4D Scene Modeling from Street View Imagescited by 35
  26. 2023Gaining Wisdom from Setbacks: Aligning Large Language Models via Mistake Analysiscited by 34
  27. 2023Empowering Autonomous Driving with Large Language Models: A Safety Perspectivecited by 34
  28. 2024SimGen: Simulator-conditioned Driving Scene Generationcited by 34
  29. 2024DreamForge: Motion-Aware Autoregressive Video Generation for Multi-View Driving Scenescited by 34
  30. 2024DrivingSphere: Building a High-fidelity 4D World for Closed-loop Simulationcited by 34
  31. 2024Deep Learning for Cross-Domain Data Fusion in Urban Computing: Taxonomy, Advances, and Outlookcited by 33
  32. 2024DetDiffusion: Synergizing Generative and Perceptive Models for Enhanced Data Generation and Perceptioncited by 32
  33. 2024SubjectDrive: Scaling Generative Data in Autonomous Driving via Subject Controlcited by 32
  34. 2024Omni-Scene: Omni-Gaussian Representation for Ego-Centric Sparse-View Scene Reconstructioncited by 32
  35. 2025Detect Anything 3D in the Wildcited by 32
  36. 2025OmniNWM: Omniscient Driving Navigation World Modelscited by 32
  37. 2024DiVE: DiT-based Video Generation with Enhanced Controlcited by 31
  38. 2023Towards Knowledge-driven Autonomous Drivingcited by 29

50 most cited of 250 All 250 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024From Bird's-Eye to Street View: Crafting Diverse and Condition-Aligned Images with Latent Diffusion Model
  2. 2025MeSS: City Mesh-Guided Outdoor Scene Generation with Cross-View Consistent Diffusion
  3. 2025Controllable 3D Outdoor Scene Generation via Scene Graphs
  4. 2026GeoNVS: Geometry Grounded Video Diffusion for Novel View Synthesis
  5. 2022JPerceiver: Joint Perception Network for Depth, Pose and Layout Estimation in Driving Scenes
  6. 2026DVGT-2: Vision-Geometry-Action Model for Autonomous Driving at Scale
  7. 2025Pointmap-Conditioned Diffusion for Consistent Novel View Synthesis
  8. 2026Joint 2D-3D Segmentation and Association in Street-level Imaging