mlsanityData to 7 Oct 2026

SGD: Street View Synthesis with Gaussian Splatting and Diffusion Prior

arXiv 2403.20079 ↗First posted 29 Mar 2024cs.CVcited by 36

On the map: World models → World models for autonomous driving

also in 1 more
Abstract (arXiv)

Novel View Synthesis (NVS) for street scenes play a critical role in the autonomous driving simulation. The current mainstream technique to achieve it is neural rendering, such as Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS). Although thrilling progress has been made, when handling street scenes, current methods struggle to maintain rendering quality at the viewpoint that deviates significantly from the training viewpoints. This issue stems from the sparse training views captured by a fixed camera on a moving vehicle. To tackle this problem, we propose a novel approach that enhances the capacity of 3DGS by leveraging prior from a Diffusion Model along with complementary multi-modal data. Specifically, we first fine-tune a Diffusion Model by adding images from adjacent frames as condition, meanwhile exploiting depth data from LiDAR point clouds to supply additional spatial information. Then we apply the Diffusion Model to regularize the 3DGS at unseen views during training. Experimental results validate the effectiveness of our method compared with current state-of-the-art models, and demonstrate its advance in rendering images from broader views.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Placed in World models for autonomous driving, one of 11 directions in World models. Builds on 53 of its 60 references that are in the map. First posted on arXiv 29 Mar 2024.

Builds on

53 of 60 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Street Gaussians: Modeling Dynamic Urban Scenes with Gaussian Splatting2 Jan
  2. 2017CARLA: An Open Urban Driving Simulator10 Nov

Other references · 51

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  5. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  6. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
  7. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  8. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  9. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  10. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,574
  11. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  12. 2023Zero-1-to-3: Zero-shot One Image to 3D Objectcited by 1,483
38 more
  1. 2021Depth-supervised NeRF: Fewer Views and Faster Training for Freecited by 732
  2. 2022Block-NeRF: Scalable Large Scene Neural View Synthesiscited by 720
  3. 2023Zip-NeRF: Anti-Aliased Grid-Based Neural Radiance Fieldscited by 667
  4. 2021Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstructioncited by 646
  5. 2023Nerfstudio: A Modular Framework for Neural Radiance Field Developmentcited by 596
  6. 2021KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3Dcited by 570
  7. 2021Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fieldscited by 569
  8. 2017AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehiclescited by 517
  9. 2023Zero123++: a Single Image to Consistent Multi-view Diffusion Base Modelcited by 516
  10. 2021Mega-NeRF: Scalable Construction of Large-Scale NeRFs for Virtual Fly-Throughscited by 396
  11. 2023DrivingGaussian: Composite Gaussian Splatting for Surrounding Dynamic Autonomous Driving Scenescited by 359
  12. 2023ReconFusion: 3D Reconstruction with Diffusion Priorscited by 317
  13. 2023FreeNeRF: Improving Few-shot Neural Rendering with Free Frequency Regularizationcited by 310
  14. 2021Dense Depth Priors for Neural Radiance Fields from Sparse Input Viewscited by 301
  15. 2020Neural Scene Graphs for Dynamic Scenescited by 298
  16. 2023UniSim: A Neural Closed-Loop Sensor Simulatorcited by 294
  17. 2023Generative Novel View Synthesis with 3D-Aware Diffusion Modelscited by 273
  18. 2021Urban Radiance Fieldscited by 263
  19. 2023SparseNeRF: Distilling Depth Ranking for Few-shot Novel View Synthesiscited by 247
  20. 2024A Survey on 3D Gaussian Splattingcited by 221
  21. 2023MVImgNet: A Large-scale Dataset of Multi-view Imagescited by 219
  22. 2022SparseFusion: Distilling View-conditioned Diffusion for 3D Reconstructioncited by 216
  23. 2023MARS: An Instance-aware, Modular and Realistic Simulator for Autonomous Drivingcited by 170
  24. 2022CLIP2Point: Transfer CLIP to Point Cloud Classification with Image-Depth Pre-trainingcited by 159
  25. 2023SUDS: Scalable Urban Dynamic Scenescited by 151
  26. 2023NeuRAD: Neural Rendering for Autonomous Drivingcited by 146
  27. 2023Periodic Vibration Gaussian: Dynamic Urban Scene Reconstruction and Real-time Renderingcited by 143
  28. 2023SparseGS: Sparse View Synthesis using 3D Gaussian Splattingcited by 134
  29. 2024Recent Advances in 3D Gaussian Splattingcited by 130
  30. 2023DiffusioNeRF: Regularizing Neural Radiance Fields with Denoising Diffusion Modelscited by 127
  31. 2023StreetSurf: Extending Multi-view Implicit Surface Reconstruction to Street Viewscited by 120
  32. 2020LGSVL Simulator: A High Fidelity Simulator for Autonomous Drivingcited by 119
  33. 2023S-NeRF: Neural Radiance Fields for Street Viewscited by 116
  34. 2023CompletionFormer: Depth Completion with Convolutions and Vision Transformerscited by 92
  35. 2023Urban Radiance Field Representation with Deformable Neural Mesh Primitivescited by 55
  36. 2023GeCoNeRF: Few-shot Neural Radiance Fields via Geometric Consistencycited by 54
  37. 2023ViP-NeRF: Visibility Prior for Sparse Input Neural Radiance Fieldscited by 43
  38. 2023Real-Time Neural Rasterization for Large Scenescited by 37

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 15; 2025: 16; 2026: 5.

36 papers.

Sort:

By area:

  1. 2024DriveDreamer4D: World Models Are Effective Data Machines for 4D Driving Scene Representationcited by 123
  2. 2024ReconDreamer: Crafting World Models for Driving Scene Reconstruction via Online Restorationcited by 89
  3. 2024HUGSIM: A Real-Time, Photo-Realistic and Closed-Loop Simulator for Autonomous Drivingcited by 84
  4. 20253D and 4D World Modeling: A Surveycited by 64
  5. 2024FreeVS: Generative View Synthesis on Free Driving Trajectorycited by 52
  6. 2024StreetCrafter: Street View Synthesis with Controllable Video Diffusion Modelscited by 49
  7. 2025ReconDreamer++: Harmonizing Generative and Reconstructive Models for Driving Scene Representationcited by 36
  8. 2024FreeSim: Toward Free-viewpoint Camera Simulation in Driving Scenescited by 30
  9. 2024GausSurf: Geometry-Guided 3D Gaussian Splatting for Surface Reconstructioncited by 19
  10. 2024GGS: Generalizable Gaussian Splatting for Lane Switching in Autonomous Drivingcited by 17
  11. 2025MTGS: Multi-Traversal Gaussian Splattingcited by 17
  12. 2025GSFix3D: Diffusion-Guided Repair of Novel Views in Gaussian Splattingcited by 17
24 more
  1. 2024Bench2Drive-R: Turning Real World Data into Reactive Closed-Loop Autonomous Driving Benchmark by Generative Modelcited by 14
  2. 2024Extrapolated Urban View Synthesis Benchmarkcited by 13
  3. 2024Driving View Synthesis on Free-form Trajectories with Generative Priorcited by 11
  4. 2025Vivid4D: Improving 4D Reconstruction from Monocular Video by Video Inpaintingcited by 11
  5. 2025Learning-based 3D Reconstruction in Autonomous Driving: A Comprehensive Surveycited by 9
  6. 2024OccFusion: Rendering Occluded Humans with Generative Diffusion Priorscited by 8
  7. 2025Generative AI for Autonomous Driving: Frontiers and Opportunitiescited by 7
  8. 2025WonderFree: Enhancing Novel View Quality and Cross-View Consistency for 3D Scene Explorationcited by 7
  9. 2025Pointmap-Conditioned Diffusion for Consistent Novel View Synthesiscited by 4
  10. 2024Radiance Fields from Photonscited by 3
  11. 2025ExtraGS: Geometric-Aware Trajectory Extrapolation with Uncertainty-Guided Generative Priorscited by 3
  12. 2026UFO: Unifying Feed-Forward and Optimization-based Methods for Large Driving Scene Modelingcited by 3
  13. 2024GPD-1: Generative Pre-training for Drivingcited by 2
  14. 2025RGE-GS: Reward-Guided Expansive Driving Scene Reconstruction via Diffusion Priorscited by 2
  15. 2026FreeFix: Boosting 3D Gaussian Splatting via Fine-Tuning-Free Diffusion Modelscited by 2
  16. 2025sshELF: Single-Shot Hierarchical Extrapolation of Latent Features for 3D Reconstruction from Sparse-Viewscited by 1
  17. 2025LidarPainter: One-Step Away From Any Lidar View To Novel Guidancecited by 1
  18. 2025FaithFusion: Harmonizing Reconstruction and Generation via Pixel-wise Information Gaincited by 1
  19. 2026DiffPlace: Street View Generation via Place-Controllable Diffusion Model Enhancing Place Recognitioncited by 1
  20. 2026Pseudo-View Enhancement via Confidence Fusion for Unposed Sparse-View Reconstructioncited by 1
  21. 2024Bootstrap-GS: Self-Supervised Augmentation for High-Fidelity Gaussian Splattingcited by 0
  22. 2025MultiEditor: Controllable Multimodal Object Editing for Driving Scenarios Using 3D Gaussian Splatting Priorscited by 0
  23. 2025Bridging Clear and Adverse Driving Conditionscited by 0
  24. 2026Leveraging Visual and Geometric Priors for Metric-scale and Complete Vehicle Gaussian Reconstruction from Limited Viewscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MuDG: Taming Multi-modal Diffusion with Gaussian Splatting for Urban Scene Reconstruction
  2. 2026Leveraging NeRF-Rendered Images for 3D Gaussian Splatting
  3. 2025Hybrid Gaussian Splatting for Novel Urban View Synthesis
  4. 2024textitS3Gaussian: Self-Supervised Street Gaussians for Autonomous Driving
  5. 2025VDEGaussian: Video Diffusion Enhanced 4D Gaussian Splatting for Dynamic Urban Scenes Modeling
  6. 2026Effective Multi-sensor Conditioning for Street-view Novel-view Synthesis
  7. 2024VEGS: View Extrapolation of Urban Scenes in 3D Gaussian Splatting using Learned Priors
  8. 2025LiDAR-GS++:Improving LiDAR Gaussian Reconstruction via Diffusion Priors