mlsanityData to 14 Oct 2026

StreetCrafter: Street View Synthesis with Controllable Video Diffusion Models

arXiv 2412.13188 ↗First posted 17 Dec 2024cs.CVcited by 49

On the map: World models → World models for autonomous driving

also in 1 more
Abstract (arXiv)

This paper aims to tackle the problem of photorealistic view synthesis from vehicle sensor data. Recent advancements in neural scene representation have achieved notable success in rendering high-quality autonomous driving scenes, but the performance significantly degrades as the viewpoint deviates from the training trajectory. To mitigate this problem, we introduce StreetCrafter, a novel controllable video diffusion model that utilizes LiDAR point cloud renderings as pixel-level conditions, which fully exploits the generative prior for novel view synthesis, while preserving precise camera control. Moreover, the utilization of pixel-level LiDAR conditions allows us to make accurate pixel-level edits to target scenes. In addition, the generative prior of StreetCrafter can be effectively incorporated into dynamic scene representations to achieve real-time rendering. Experiments on Waymo Open Dataset and PandaSet demonstrate that our model enables flexible control over viewpoint changes, enlarging the view synthesis regions for satisfying rendering, which outperforms existing methods.

Key facts. Cited by 49 papers in the map (data through 14 Oct 2026). Placed in World models for autonomous driving, one of 11 directions in World models. Builds on 75 of its 89 references that are in the map. First posted on arXiv 17 Dec 2024.

Builds on

75 of 89 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024DriveDreamer4D: World Models Are Effective Data Machines for 4D Driving Scene Representation17 Oct
  2. 2024OmniRe: Omni Urban Scene Reconstruction29 Aug
  3. 2024Dynamic 3D Gaussian Fields for Urban Areas5 Jun
  4. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability27 May
  5. 2024MagicDrive3D: Controllable 3D Generation for Any-View Rendering in Street Scenes23 May
  6. 2024SGD: Street View Synthesis with Gaussian Splatting and Diffusion Prior29 Mar
  7. 2024Street Gaussians: Modeling Dynamic Urban Scenes with Gaussian Splatting2 Jan
  8. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Driving18 Sep

Other references · 67

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  5. 2020Denoising Diffusion Implicit Modelscited by 9,781
  6. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  7. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,226
  8. 2022Scalable Diffusion Models with Transformerscited by 6,054
  9. 2022Classifier-Free Diffusion Guidancecited by 5,847
  10. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  11. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,723
  12. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,045
38 more
  1. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,025
  2. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
  3. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,451
  4. 2022Video Diffusion Modelscited by 2,305
  5. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
  6. 2023DUSt3R: Geometric 3D Vision Made Easycited by 1,567
  7. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,476
  8. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,290
  9. 2022Block-NeRF: Scalable Large Scene Neural View Synthesiscited by 724
  10. 2023Wonder3D: Single Image to 3D using Cross-Domain Diffusioncited by 689
  11. 2023Mip-Splatting: Alias-free 3D Gaussian Splattingcited by 667
  12. 2023ModelScope Text-to-Video Technical Reportcited by 658
  13. 2023MotionCtrl: A Unified and Flexible Motion Controller for Video Generationcited by 580
  14. 2023One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimizationcited by 560
  15. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Modelscited by 465
  16. 2024CameraCtrl: Enabling Camera Control for Text-to-Video Generationcited by 460
  17. 2024CAT3D: Create Anything in 3D with Multi-View Diffusion Modelscited by 432
  18. 2024ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesiscited by 403
  19. 2023Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Modelcited by 378
  20. 2024SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusioncited by 364
  21. 2023DrivingGaussian: Composite Gaussian Splatting for Surrounding Dynamic Autonomous Driving Scenescited by 360
  22. 2023LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Modelscited by 349
  23. 2021Light Field Networks: Neural Scene Representations with Single-Evaluation Renderingcited by 332
  24. 2023ReconFusion: 3D Reconstruction with Diffusion Priorscited by 318
  25. 2020Neural Scene Graphs for Dynamic Scenescited by 298
  26. 2023UniSim: A Neural Closed-Loop Sensor Simulatorcited by 297
  27. 2023MagicDrive: Street View Generation with Diverse 3D Geometry Controlcited by 252
  28. 2024Gaussian Opacity Fields: Efficient Adaptive Surface Reconstruction in Unbounded Scenescited by 232
  29. 2023EmerNeRF: Emergent Spatial-Temporal Scene Decomposition via Self-Supervisioncited by 217
  30. 2021PandaSet: Advanced Sensor Suite Dataset for Autonomous Drivingcited by 190
  31. 2024LVSM: A Large View Synthesis Model with Minimal 3D Inductive Biascited by 189
  32. 2023MARS: An Instance-aware, Modular and Realistic Simulator for Autonomous Drivingcited by 170
  33. 2024CamCo: Camera-Controllable 3D-Consistent Image-to-Video Generationcited by 166
  34. 2024VD3D: Taming Large Video Diffusion Transformers for 3D Camera Controlcited by 156
  35. 2023SUDS: Scalable Urban Dynamic Scenescited by 151
  36. 2023NeuRAD: Neural Rendering for Autonomous Drivingcited by 150
  37. 2023Periodic Vibration Gaussian: Dynamic Urban Scene Reconstruction and Real-time Renderingcited by 143
  38. 2024ReconX: Reconstruct Any Scene from Sparse Views with Video Diffusion Modelcited by 143

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 27; 2026: 22.

49 papers.

Sort:

By area:

  1. 2025End-to-End Driving with Online Trajectory Evaluation via BEV World Modelcited by 151
  2. 2025TrajectoryCrafter: Redirecting Camera Trajectory for Monocular Videos via Diffusion Modelscited by 125
  3. 20253D and 4D World Modeling: A Surveycited by 67
  4. 2025Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillationcited by 41
  5. 2025DiST-4D: Disentangled Spatiotemporal Diffusion with Metric Depth for 4D Driving Scene Generationcited by 40
  6. 2025ReconDreamer++: Harmonizing Generative and Reconstructive Models for Driving Scene Representationcited by 36
  7. 20253D Scene Generation: A Surveycited by 24
  8. 2025X-Scene: Large-Scale Driving Scene Generation with High Fidelity and Flexible Controllabilitycited by 20
  9. 2025MTGS: Multi-Traversal Gaussian Splattingcited by 19
  10. 2025ORV: 4D Occupancy-centric Robot Video Generationcited by 17
  11. 2025Challenger: Affordable Adversarial Driving Video Generationcited by 16
  12. 2026AnyView: Synthesizing Any Novel View in Dynamic Scenescited by 12
37 more
  1. 2025UniSplat: Unified Spatio-Temporal Fusion via 3D Latent Scaffolds for Dynamic Driving Scene Reconstructioncited by 10
  2. 2025MuDG: Taming Multi-modal Diffusion with Gaussian Splatting for Urban Scene Reconstructioncited by 8
  3. 2026From Rays to Projections: Better Inputs for Feed-Forward View Synthesiscited by 5
  4. 2025ExtraGS: Geometric-Aware Trajectory Extrapolation with Uncertainty-Guided Generative Priorscited by 4
  5. 2026Pantheon360: Taming Digital Twin Generation via 3D-Aware 360deg Video Diffusioncited by 4
  6. 2026HorizonForge: Driving Scene Editing with Any Trajectories and Any Vehiclescited by 3
  7. 2026Instant NuRec: Feed-Forward 3D Gaussian Reconstruction for Driving Scene Simulationcited by 3
  8. 2025RGE-GS: Reward-Guided Expansive Driving Scene Reconstruction via Diffusion Priorscited by 2
  9. 2025Causal-Entity Reflected Egocentric Traffic Accident Video Synthesiscited by 2
  10. 2025DiffVL: Diffusion-Based Visual Localization on 2D Maps via BEV-Conditioned GPS Denoisingcited by 2
  11. 2025SymDrive: Realistic and Controllable Driving Simulator via Symmetric Auto-regressive Online Restorationcited by 2
  12. 2026FreeFix: Boosting 3D Gaussian Splatting via Fine-Tuning-Free Diffusion Modelscited by 2
  13. 2026DCARL: A Divide-and-Conquer Framework for Autoregressive Long-Trajectory Video Generationcited by 2
  14. 2025FaithFusion: Harmonizing Reconstruction and Generation via Pixel-wise Information Gaincited by 1
  15. 2025ReCamDriving: LiDAR-Free Camera-Controlled Video Synthesis for Novel Trajectoriescited by 1
  16. 2026DiffPlace: Street View Generation via Place-Controllable Diffusion Model Enhancing Place Recognitioncited by 1
  17. 2026Map2World: Segment Map Conditioned Text to 3D World Generationcited by 1
  18. 2026Diffusion-guided Generalizable Enhancer for Urban Scene Reconstructioncited by 1
  19. 2026Envision4D: Envisioning Visual Futures via Feed-forward 4D Gaussian Splatting for Autonomous Drivingcited by 1
  20. 2025ADGaussian: Generalizable Gaussian Splatting for Autonomous Driving via Multi-modal Joint Learningcited by 0
  21. 2025SatDreamer360: Multiview-Consistent Generation of Ground-Level Scenes from Satellite Imagerycited by 0
  22. 2025MultiEditor: Controllable Multimodal Object Editing for Driving Scenarios Using 3D Gaussian Splatting Priorscited by 0
  23. 2025I2V-GS: Infrastructure-to-Vehicle View Transformation with Gaussian Splatting for Autonomous Driving Data Generationcited by 0
  24. 2025ArbiViewGen: Controllable Arbitrary Viewpoint Camera Data Generation for Autonomous Driving via Stable Diffusion Modelscited by 0
  25. 2025ViSE: A Systematic Approach to Vision-Only Street-View Extrapolationcited by 0
  26. 2025SCPainter: A Unified Framework for Realistic 3D Asset Insertion and Novel View Synthesiscited by 0
  27. 2026S2D: Sparse to Dense Lifting for 3D Reconstruction with Minimal Inputscited by 0
  28. 2026DriveFix: Spatio-Temporally Coherent Driving Scene Restorationcited by 0
  29. 2026ReinDriveGen: Reinforcement Post-Training for Out-of-Distribution Driving Scene Generationcited by 0
  30. 2026ConFixGS: Learning to Fix Feedforward 3D Gaussian Splatting with Confidence-Aware Diffusion Priors in Driving Scenescited by 0
  31. 2026Effective Multi-sensor Conditioning for Street-view Novel-view Synthesiscited by 0
  32. 2026WarpHammer: Densifying Scene Warps with 3D Object Priors for Extreme View Synthesiscited by 0
  33. 2026DriveWeaver: Point-Conditioned Video Inpainting for Controllable Vehicle Insertion in Autonomous Driving Simulationcited by 0
  34. 2026SPVC: Structured and Panoptic Video Fixing for Cross-Dataset Driving Scene Renderingcited by 0
  35. 2026PRG-Fusion: Orchestrating Generative Priors with Reconstruction Evidence for Driving View Synthesiscited by 0
  36. 2026StreetDiff: Multi-view Street Scenes Generation via Cross-view Consistent Multi-view Stable Diffusion with Structure Promptscited by 0
  37. 2026A Survey on End-to-End Autonomous Driving Training from the Perspectives of Data, Strategy, and Platformcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Pointmap-Conditioned Diffusion for Consistent Novel View Synthesis
  2. 2025LidarPainter: One-Step Away From Any Lidar View To Novel Guidance
  3. 2025SceneCrafter: Controllable Multi-View Driving Scene Editing
  4. 2025DrivingGaussian++: Towards Realistic Reconstruction and Editable Simulation for Surrounding Dynamic Driving Scenes
  5. 2025MoVieDrive: Urban Scene Synthesis with Multi-Modal Multi-View Video Diffusion Transformer
  6. 2024SyntheOcc: Synthesize Geometric-Controlled Street View Images through 3D Semantic MPIs
  7. 2025MeSS: City Mesh-Guided Outdoor Scene Generation with Cross-View Consistent Diffusion
  8. 2024Drive-1-to-3: Enriching Diffusion Priors for Novel View Synthesis of Real Vehicles