mlsanityData to 7 Oct 2026

Street Gaussians: Modeling Dynamic Urban Scenes with Gaussian Splatting

arXiv 2401.01339 ↗First posted 2 Jan 2024cs.CV cs.GRcited by 218

On the map: World models → World models for autonomous driving

Abstract (arXiv)

This paper aims to tackle the problem of modeling dynamic urban streets for autonomous driving scenes. Recent methods extend NeRF by incorporating tracked vehicle poses to animate vehicles, enabling photo-realistic view synthesis of dynamic urban street scenes. However, significant limitations are their slow training and rendering speed. We introduce Street Gaussians, a new explicit scene representation that tackles these limitations. Specifically, the dynamic urban scene is represented as a set of point clouds equipped with semantic logits and 3D Gaussians, each associated with either a foreground vehicle or the background. To model the dynamics of foreground object vehicles, each object point cloud is optimized with optimizable tracked poses, along with a 4D spherical harmonics model for the dynamic appearance. The explicit representation allows easy composition of object vehicles and background, which in turn allows for scene editing operations and rendering at 135 FPS (1066 × 1600 resolution) within half an hour of training. The proposed method is evaluated on multiple challenging benchmarks, including KITTI and Waymo Open datasets. Experiments show that the proposed method consistently outperforms state-of-the-art methods across all datasets. The code will be released to ensure reproducibility.

Key facts. Cited by 218 papers in the map (data through 7 Oct 2026). Placed in World models for autonomous driving, one of 11 directions in World models. Builds on 67 of its 80 references that are in the map. First posted on arXiv 2 Jan 2024.

Builds on

67 of 80 references.

Closest ancestors · 1

Same direction, newest first

  1. 2017CARLA: An Open Urban Driving Simulator10 Nov

Other references · 66

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  3. 2023Segment Anythingcited by 8,158
  4. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
  5. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
  6. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  7. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,436
  8. 2021Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fieldscited by 2,013
  9. 2021Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fieldscited by 1,957
  10. 20234D Gaussian Splatting for Real-Time Dynamic Scene Renderingcited by 1,144
  11. 2020GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fieldscited by 784
  12. 2023Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstructioncited by 753
38 more
  1. 2020Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenescited by 750
  2. 2023K-Planes: Explicit Radiance Fields in Space, Time, and Appearancecited by 742
  3. 2023Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesiscited by 731
  4. 2022Block-NeRF: Scalable Large Scene Neural View Synthesiscited by 720
  5. 2023Zip-NeRF: Anti-Aliased Grid-Based Neural Radiance Fieldscited by 667
  6. 2023Nerfstudio: A Modular Framework for Neural Radiance Field Developmentcited by 596
  7. 2017AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehiclescited by 517
  8. 2023Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splattingcited by 470
  9. 2021Mega-NeRF: Scalable Construction of Large-Scale NeRFs for Virtual Fly-Throughscited by 396
  10. 2023DrivingGaussian: Composite Gaussian Splatting for Surrounding Dynamic Autonomous Driving Scenescited by 359
  11. 2020Virtual KITTI 2cited by 345
  12. 2019Neural Point-Based Graphicscited by 326
  13. 2020Neural Scene Graphs for Dynamic Scenescited by 298
  14. 2023UniSim: A Neural Closed-Loop Sensor Simulatorcited by 294
  15. 2021Urban Radiance Fieldscited by 263
  16. 2022Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representationcited by 243
  17. 2023EmerNeRF: Emergent Spatial-Temporal Scene Decomposition via Self-Supervisioncited by 216
  18. 2021Point-Based Neural Rendering with Per-View Optimizationcited by 188
  19. 2023HyperReel: High-Fidelity 6-DoF Video with Ray-Conditioned Samplingcited by 173
  20. 2022Panoptic Lifting for 3D Scene Understanding with Neural Fieldscited by 171
  21. 2023MARS: An Instance-aware, Modular and Realistic Simulator for Autonomous Drivingcited by 170
  22. 2022Panoptic NeRF: 3D-to-2D Label Transfer for Panoptic Urban Scene Segmentationcited by 152
  23. 2023SUDS: Scalable Urban Dynamic Scenescited by 151
  24. 2021Efficient Neural Radiance Fields for Interactive Free-viewpoint Videocited by 150
  25. 2023NeuRAD: Neural Rendering for Autonomous Drivingcited by 146
  26. 2023Periodic Vibration Gaussian: Dynamic Urban Scene Reconstruction and Real-time Renderingcited by 143
  27. 2020LiDARsim: Realistic LiDAR Simulation by Leveraging the Real Worldcited by 137
  28. 2023StreetSurf: Extending Multi-view Implicit Surface Reconstruction to Street Viewscited by 120
  29. 2023S-NeRF: Neural Radiance Fields for Street Viewscited by 116
  30. 2023Grid-guided Neural Radiance Fields for Large Urban Scenescited by 99
  31. 20234K4D: Real-Time 4D View Synthesis at 4K Resolutioncited by 91
  32. 2023Neural Fields meet Explicit Geometric Representation for Inverse Rendering of Urban Scenescited by 89
  33. 2022Differentiable Point-Based Radiance Fields for Efficient View Synthesiscited by 86
  34. 2022Video K-Net: A Simple, Strong, and Unified Baseline for Video Segmentationcited by 79
  35. 2021GeoSim: Realistic Video Simulation via Geometry-Aware Composition for Self-Drivingcited by 75
  36. 2020SurfelGAN: Synthesizing Realistic Sensor Data for Autonomous Drivingcited by 68
  37. 2023Neural LiDAR Fields for Novel View Synthesiscited by 65
  38. 2022DisCoScene: Spatially Disentangled Generative Radiance Fields for Controllable 3D-aware Scene Synthesiscited by 61

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 35; 2025: 92; 2026: 91.

218 papers.

Sort:

By area:

  1. 2025Pseudo-Simulation for Autonomous Drivingcited by 141
  2. 2024textitS3Gaussian: Self-Supervised Street Gaussians for Autonomous Drivingcited by 105
  3. 2024GaussianImage: 1000 FPS Image Representation and Compression by 2D Gaussian Splattingcited by 97
  4. 2024HUGSIM: A Real-Time, Photo-Realistic and Closed-Loop Simulator for Autonomous Drivingcited by 84
  5. 2025RAD: Training an End-to-End Driving Policy via Large-Scale 3DGS-based Reinforcement Learningcited by 76
  6. 20253D and 4D World Modeling: A Surveycited by 64
  7. 2024DrivingForward: Feed-forward 3D Gaussian Splatting for Driving Scene Reconstruction from Flexible Surround-view Inputcited by 53
  8. 2024FreeVS: Generative View Synthesis on Free Driving Trajectorycited by 52
  9. 2024StreetCrafter: Street View Synthesis with Controllable Video Diffusion Modelscited by 49
  10. 2025DriveLaW:Unifying Planning and Video Generation in a Latent Driving Worldcited by 43
  11. 2024SemGauss-SLAM: Dense Semantic Gaussian Splatting SLAMcited by 41
  12. 2025DiST-4D: Disentangled Spatiotemporal Diffusion with Metric Depth for 4D Driving Scene Generationcited by 39
38 more
  1. 2024SGD: Street View Synthesis with Gaussian Splatting and Diffusion Priorcited by 36
  2. 2025SimScale: Learning to Drive via Real-World Simulation at Scalecited by 34
  3. 2025WonderTurbo: Generating Interactive 3D World in 0.72 Secondscited by 31
  4. 2025What Is The Best 3D Scene Representation for Robotics? From Geometric to Foundation Modelscited by 29
  5. 2025CityGS-X: A Scalable Architecture for Efficient and Geometrically Accurate Large-Scale Scene Reconstructioncited by 26
  6. 2024Forging Vision Foundation Models for Autonomous Driving: Challenges, Methodologies, and Opportunitiescited by 24
  7. 2024GauStudio: A Modular Framework for 3D Gaussian Splatting and Beyondcited by 24
  8. 2025GaussianFlowOcc: Sparse and Weakly Supervised Occupancy Estimation using Gaussian Splatting and Temporal Flowcited by 24
  9. 2025GaussianDWM: 3D Gaussian Driving World Model for Unified Scene Understanding and Multi-Modal Generationcited by 23
  10. 2024GauU-Scene V2: Assessing the Reliability of Image-Based Metrics with Expansive Lidar Image Dataset Using 3DGS and NeRFcited by 21
  11. 2025X-Scene: Large-Scale Driving Scene Generation with High Fidelity and Flexible Controllabilitycited by 20
  12. 2024Neural Radiance Field in Autonomous Driving: A Surveycited by 19
  13. 2024Segment Any 4D Gaussianscited by 19
  14. 2024StreetSurfGS: Scalable Urban Street Surface Reconstruction with Planar-based Gaussian Splattingcited by 19
  15. 2024GausSurf: Geometry-Guided 3D Gaussian Splatting for Surface Reconstructioncited by 19
  16. 2025GeoDrive: 3D Geometry-Informed Driving World Model with Precise Action Controlcited by 19
  17. 2024Stag-1: Towards Realistic 4D Driving Simulation with Video Generation Modelcited by 18
  18. 2025MTGS: Multi-Traversal Gaussian Splattingcited by 17
  19. 2025AD-GS: Object-Aware B-Spline Gaussian Splatting for Self-Supervised Autonomous Drivingcited by 17
  20. 2025PRIX: Learning to Plan from Raw Pixels for End-to-End Autonomous Drivingcited by 16
  21. 2024GauU-Scene: A Scene Reconstruction Benchmark on Large Scale 3D Reconstruction Dataset Using Gaussian Splattingcited by 15
  22. 20243DGS-ReLoc: 3D Gaussian Splatting for Map Representation and Visual ReLocalizationcited by 15
  23. 2025GeoPredict: Leveraging Predictive Kinematics and 3D Gaussian Geometry for Precise VLA Manipulationcited by 15
  24. 2024Bench2Drive-R: Turning Real World Data into Reactive Closed-Loop Autonomous Driving Benchmark by Generative Modelcited by 14
  25. 2025GaussianFormer3D: Multi-Modal Gaussian-based Semantic Occupancy Prediction with 3D Deformable Attentioncited by 14
  26. 2025Unifying Appearance Codes and Bilateral Grids for Driving Scene Gaussian Splattingcited by 14
  27. 2025DriveGen3D: Boosting Feed-Forward Driving Scene Generation with Efficient Video Diffusioncited by 14
  28. 2024EMD: Explicit Motion Modeling for High-Quality Street Gaussian Splattingcited by 13
  29. 2024TCLC-GS: Tightly Coupled LiDAR-Camera Gaussian Splatting for Autonomous Drivingcited by 11
  30. 2024Driving View Synthesis on Free-form Trajectories with Generative Priorcited by 11
  31. 2024CoSurfGS:Collaborative 3D Surface Gaussian Splatting with Distributed Learning for Large Scene Reconstructioncited by 11
  32. 2025HAIF-GS: Hierarchical and Induced Flow-Guided Gaussian Splatting for Dynamic Scenecited by 11
  33. 2025WorldSplat: Gaussian-Centric Feed-Forward 4D Scene Generation for Autonomous Drivingcited by 11
  34. 2026NTIRE 2026 3D Restoration and Reconstruction in Real-world Adverse Conditions: RealX3D Challenge Resultscited by 11
  35. 2024SweepEvGS: Event-Based 3D Gaussian Splatting for Macro and Micro Radiance Field Rendering from a Single Sweepcited by 10
  36. 20253R-GS: Best Practice in Optimizing Camera Poses Along with 3DGScited by 10
  37. 2025UniUGP: Unifying Understanding, Generation, and Planing For End-to-end Autonomous Drivingcited by 10
  38. 2025Learning-based 3D Reconstruction in Autonomous Driving: A Comprehensive Surveycited by 9

50 most cited of 218 All 218 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024DeSiRe-GS: 4D Street Gaussians for Static-Dynamic Decomposition and Surface Reconstruction for Urban Driving Scenes
  2. 2024UniGaussian: Driving Scene Reconstruction from Multiple Camera Models via Unified Gaussian Representations
  3. 2025Hybrid Gaussian Splatting for Novel Urban View Synthesis
  4. 2024Dynamic 3D Gaussian Fields for Urban Areas
  5. 2025StyledStreets: Multi-style Street Simulator with Spatial and Temporal Consistency
  6. 2025CoDa-4DGS: Dynamic Gaussian Splatting with Context and Deformation Awareness for Autonomous Driving
  7. 2024GaussianAD: Gaussian-Centric End-to-End Autonomous Driving
  8. 2025VDEGaussian: Video Diffusion Enhanced 4D Gaussian Splatting for Dynamic Urban Scenes Modeling