mlsanityData to 7 Oct 2026

HUGS: Holistic Urban 3D Scene Understanding via Gaussian Splatting

arXiv 2403.12722 ↗First posted 19 Mar 2024cs.CVcited by 141

Not on the map. Why?

Abstract (arXiv)

Holistic understanding of urban scenes based on RGB images is a challenging yet important problem. It encompasses understanding both the geometry and appearance to enable novel view synthesis, parsing semantic labels, and tracking moving objects. Despite considerable progress, existing approaches often focus on specific aspects of this task and require additional inputs such as LiDAR scans or manually annotated 3D bounding boxes. In this paper, we introduce a novel pipeline that utilizes 3D Gaussian Splatting for holistic urban scene understanding. Our main idea involves the joint optimization of geometry, appearance, semantics, and motion using a combination of static and dynamic 3D Gaussians, where moving object poses are regularized via physical constraints. Our approach offers the ability to render new viewpoints in real-time, yielding 2D and 3D semantic information with high accuracy, and reconstruct dynamic scenes, even in scenarios where 3D bounding box detection are highly noisy. Experimental results on KITTI, KITTI-360, and Virtual KITTI 2 demonstrate the effectiveness of our approach.

Key facts. Cited by 141 papers in the map (data through 7 Oct 2026). Builds on 37 of its 52 references that are in the map. First posted on arXiv 19 Mar 2024.

Builds on

37 of 52 references.

References · 37

  1. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  2. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
  3. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  4. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
  5. 2021Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fieldscited by 2,013
  6. 2021Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fieldscited by 1,957
  7. 2020NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collectionscited by 1,230
  8. 2023Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstructioncited by 753
  9. 2023Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesiscited by 731
  10. 2022Block-NeRF: Scalable Large Scene Neural View Synthesiscited by 720
  11. 2023Zip-NeRF: Anti-Aliased Grid-Based Neural Radiance Fieldscited by 667
  12. 2023Nerfstudio: A Modular Framework for Neural Radiance Field Developmentcited by 596
25 more
  1. 2021KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3Dcited by 570
  2. 2023Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splattingcited by 470
  3. 2022MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstructioncited by 467
  4. 2021In-Place Scene Labelling and Understanding with Implicit Scene Representationcited by 411
  5. 2019Panoptic-DeepLabcited by 362
  6. 2021Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets from 3D Scanscited by 336
  7. 2020Neural Scene Graphs for Dynamic Scenescited by 298
  8. 2023UniSim: A Neural Closed-Loop Sensor Simulatorcited by 294
  9. 2022Unifying Flow, Stereo and Depth Estimationcited by 270
  10. 2022Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representationcited by 243
  11. 2021Learning Object-Compositional Neural Radiance Field for Editable Scene Renderingcited by 228
  12. 2023EmerNeRF: Emergent Spatial-Temporal Scene Decomposition via Self-Supervisioncited by 216
  13. 2020Hierarchical Multi-Scale Attention for Semantic Segmentationcited by 183
  14. 2023MARS: An Instance-aware, Modular and Realistic Simulator for Autonomous Drivingcited by 170
  15. 2022Panoptic NeRF: 3D-to-2D Label Transfer for Panoptic Urban Scene Segmentationcited by 152
  16. 2023StreetSurf: Extending Multi-view Implicit Surface Reconstruction to Street Viewscited by 120
  17. 2021NeSF: Neural Semantic Fields for Generalizable Semantic Segmentation of 3D Scenescited by 119
  18. 2020Category Level Object Pose Estimation via Neural Analysis-by-Synthesiscited by 100
  19. 20234K4D: Real-Time 4D View Synthesis at 4K Resolutioncited by 91
  20. 2021Monocular Quasi-Dense 3D Object Trackingcited by 80
  21. 2023iDisc: Internal Discretization for Monocular Depth Estimationcited by 71
  22. 2021InverseForm: A Loss Function for Structured Boundary-Aware Segmentationcited by 65
  23. 2023Behind the Scenes: Density Fields for Single View Reconstructioncited by 63
  24. 2023Urban Radiance Field Representation with Deformable Neural Mesh Primitivescited by 55
  25. 2022Learning Multi-View Aggregation In the Wild for Large-Scale 3D Semantic Segmentationcited by 47

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 37; 2025: 70; 2026: 34.

141 papers.

Sort:

By area:

  1. 2024OmniRe: Omni Urban Scene Reconstructioncited by 159
  2. 2024OpenGaussian: Towards Point-Level 3D Gaussian-based Open Vocabulary Understandingcited by 141
  3. 2024DriveDreamer4D: World Models Are Effective Data Machines for 4D Driving Scene Representationcited by 123
  4. 2024UniScene: Unified Occupancy-centric Driving Scene Generationcited by 102
  5. 2024ReconDreamer: Crafting World Models for Driving Scene Reconstruction via Online Restorationcited by 89
  6. 2024HUGSIM: A Real-Time, Photo-Realistic and Closed-Loop Simulator for Autonomous Drivingcited by 84
  7. 20243D Gaussian Splatting: Survey, Technologies, Challenges, and Opportunitiescited by 77
  8. 2024SplatAD: Real-Time Lidar and Camera Rendering with 3D Gaussian Splatting for Autonomous Drivingcited by 66
  9. 2024Wild-GS: Real-Time Novel View Synthesis from Unconstrained Photo Collectionscited by 56
  10. 2024STORM: Spatio-Temporal Reconstruction Model for Large-Scale Outdoor Scenescited by 53
  11. 2024FreeVS: Generative View Synthesis on Free Driving Trajectorycited by 52
  12. 20243D Convex Splatting: Radiance Field Rendering with 3D Smooth Convexescited by 50
38 more
  1. 2025Triangle Splatting for Real-Time Radiance Field Renderingcited by 50
  2. 2024DeSiRe-GS: 4D Street Gaussians for Static-Dynamic Decomposition and Surface Reconstruction for Urban Driving Scenescited by 45
  3. 2024Dynamic 3D Gaussian Fields for Urban Areascited by 35
  4. 20254D LangSplat: 4D Language Gaussian Splatting via Multimodal Large Language Modelscited by 35
  5. 20243D Vision-Language Gaussian Splattingcited by 30
  6. 2024FreeSim: Toward Free-viewpoint Camera Simulation in Driving Scenescited by 30
  7. 2025Scenario Dreamer: Vectorized Latent Diffusion for Generating Driving Simulation Environmentscited by 27
  8. 2025StarGen: A Spatiotemporal Autoregression Framework with Video Diffusion Model for Scalable and Controllable Scene Generationcited by 26
  9. 2025CoDa-4DGS: Dynamic Gaussian Splatting with Context and Deformation Awareness for Autonomous Drivingcited by 25
  10. 2025Embodied AI: From LLMs to World Modelscited by 23
  11. 2024SplatFlow: Self-Supervised Dynamic Gaussian Splatting in Neural Motion Flow Field for Autonomous Drivingcited by 20
  12. 2025EVolSplat: Efficient Volume-based Gaussian Splatting for Urban View Synthesiscited by 20
  13. 2025ReferSplat: Referring Segmentation in 3D Gaussian Splattingcited by 20
  14. 2024Neural Radiance Field in Autonomous Driving: A Surveycited by 19
  15. 2025Describe Anything Anywhere At Any Momentcited by 19
  16. 2025MTGS: Multi-Traversal Gaussian Splattingcited by 17
  17. 2025AD-GS: Object-Aware B-Spline Gaussian Splatting for Self-Supervised Autonomous Drivingcited by 17
  18. 2024DC-Gaussian: Improving 3D Gaussian Splatting for Reflective Dash Cam Videoscited by 16
  19. 2025PanoGS: Gaussian-based Panoptic Segmentation for 3D Open Vocabulary Scene Understandingcited by 16
  20. 2024VisionPAD: A Vision-Centric Pre-training Paradigm for Autonomous Drivingcited by 15
  21. 2024DeSplat: Decomposed Gaussian Splatting for Distractor-Free Renderingcited by 15
  22. 20244D Gaussian Splatting: Modeling Dynamic Scenes with Native 4D Primitivescited by 15
  23. 2025Unifying Appearance Codes and Bilateral Grids for Driving Scene Gaussian Splattingcited by 14
  24. 2025DriveGen3D: Boosting Feed-Forward Driving Scene Generation with Efficient Video Diffusioncited by 14
  25. 2024EMD: Explicit Motion Modeling for High-Quality Street Gaussian Splattingcited by 13
  26. 2024Extrapolated Urban View Synthesis Benchmarkcited by 13
  27. 2025RoGSplat: Learning Robust Generalizable Human Gaussian Splatting from Sparse Multi-View Imagescited by 12
  28. 2025Vivid4D: Improving 4D Reconstruction from Monocular Video by Video Inpaintingcited by 11
  29. 2025CRUISE: Cooperative Reconstruction and Editing in V2X Scenarios using Gaussian Splattingcited by 11
  30. 2025GS-Occ3D: Scaling Vision-only Occupancy Reconstruction with Gaussian Splattingcited by 11
  31. 2025WorldSplat: Gaussian-Centric Feed-Forward 4D Scene Generation for Autonomous Drivingcited by 11
  32. 2025See4D: Pose-Free 4D Generation via Auto-Regressive Video Inpaintingcited by 11
  33. 2024UniGaussian: Driving Scene Reconstruction from Multiple Camera Models via Unified Gaussian Representationscited by 10
  34. 2024LiDAR-RT: Gaussian-based Ray Tracing for Dynamic LiDAR Re-simulationcited by 10
  35. 2025Micro-macro Wavelet-based Gaussian Splatting for 3D Reconstruction from Unconstrained Imagescited by 10
  36. 2025RealEngine: Simulating Autonomous Driving in Realistic Contextcited by 10
  37. 2025RobustSplat: Decoupling Densification and Dynamics for Transient-Free 3DGScited by 10
  38. 20253DGabSplat: 3D Gabor Splatting for Frequency-adaptive Radiance Field Renderingcited by 10

50 most cited of 141 All 141 (JSON)