# ProSGNeRF: Progressive Dynamic Neural Scene Graph with Frequency Modulated Foundation Model in Urban Scenes

arXiv: https://arxiv.org/abs/2312.09076 · HTML: https://mlsanity.com/paper/2312.09076/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 58 references that are in the map.
- First posted on arXiv 14 Dec 2023.

## Builds on (most cited first, 30 of 37)

- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields](https://mlsanity.com/paper/2111.12077/) (2021)
- [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021)
- [NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections](https://mlsanity.com/paper/2008.02268/) (2020)
- [Nerfies: Deformable Neural Radiance Fields](https://mlsanity.com/paper/2011.12948/) (2020)
- [4D Gaussian Splatting for Real-Time Dynamic Scene Rendering](https://mlsanity.com/paper/2310.08528/) (2023)
- [NeRF++: Analyzing and Improving Neural Radiance Fields](https://mlsanity.com/paper/2010.07492/) (2020)
- [Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes](https://mlsanity.com/paper/2011.13084/) (2020)
- [Depth-supervised NeRF: Fewer Views and Faster Training for Free](https://mlsanity.com/paper/2107.02791/) (2021)
- [Block-NeRF: Scalable Large Scene Neural View Synthesis](https://mlsanity.com/paper/2202.05263/) (2022)
- [Dynamic View Synthesis from Dynamic Monocular Video](https://mlsanity.com/paper/2105.06468/) (2021)
- [Space-time Neural Irradiance Fields for Free-Viewpoint Video](https://mlsanity.com/paper/2011.12950/) (2020)
- [Mega-NeRF: Scalable Construction of Large-Scale NeRFs for Virtual Fly-Throughs](https://mlsanity.com/paper/2112.10703/) (2021)
- [DrivingGaussian: Composite Gaussian Splatting for Surrounding Dynamic Autonomous Driving Scenes](https://mlsanity.com/paper/2312.07920/) (2023)
- [Virtual KITTI 2](https://mlsanity.com/paper/2001.10773/) (2020)
- [FreeNeRF: Improving Few-shot Neural Rendering with Free Frequency Regularization](https://mlsanity.com/paper/2303.07418/) (2023)
- [Neural Scene Graphs for Dynamic Scenes](https://mlsanity.com/paper/2011.10379/) (2020)
- [Urban Radiance Fields](https://mlsanity.com/paper/2111.14643/) (2021)
- [Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representation](https://mlsanity.com/paper/2205.04334/) (2022)
- [BungeeNeRF: Progressive Neural Radiance Field for Extreme Multi-scale Scene Rendering](https://mlsanity.com/paper/2112.05504/) (2021)
- [EmerNeRF: Emergent Spatial-Temporal Scene Decomposition via Self-Supervision](https://mlsanity.com/paper/2311.02077/) (2023)
- [HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields](https://mlsanity.com/paper/2106.13228/) (2021)
- [MARS: An Instance-aware, Modular and Realistic Simulator for Autonomous Driving](https://mlsanity.com/paper/2307.15058/) (2023)
- [Panoptic NeRF: 3D-to-2D Label Transfer for Panoptic Urban Scene Segmentation](https://mlsanity.com/paper/2203.15224/) (2022)

## Built on by (most cited first, 3 of 3)

- [LLM-Augmented Traffic Signal Control with LSTM-Based Traffic State Prediction and Safety-Constrained Decision Support](https://mlsanity.com/paper/2604.23902/) (2026)
- [Multi-Submap Implicit Neural SLAM with Local-to-Global Loop Closure for Large-Scale Scene Reconstruction](https://mlsanity.com/paper/2608.09146/) (2026)
- [BehaviorWorldGen: Closing the Loop between Action Models and World Simulators via Controllable Behavior-Aware Structured World Generation](https://mlsanity.com/paper/2608.22187/) (2026)

## Abstract (arXiv)

Implicit neural representation has demonstrated promising results in 3D reconstruction on various scenes. However, existing approaches either struggle to model fast-moving objects or are incapable of handling large-scale camera ego-motions in urban environments. This leads to low-quality synthesized views of the large-scale urban scenes. In this paper, we aim to jointly solve the problems caused by large-scale scenes and fast-moving vehicles, which are more practical and challenging. To this end, we propose a progressive scene graph network architecture to learn the local scene representations of dynamic objects and global urban scenes. The progressive learning architecture dynamically allocates a new local scene graph trained on frames within a temporal window, with the window size automatically determined, allowing us to scale up the representation to arbitrarily large scenes. Besides, according to our observations, the training views of dynamic objects are relatively sparse according to rapid movements, which leads to a significant decline in reconstruction accuracy for dynamic objects. Therefore, we utilize a foundation model network to encode the latent code. Specifically, we leverage the generalization capability of the visual foundation model DINOv2 to extract appearance and shape codes, and train the network on a large-scale urban scene object dataset to enhance its prior modeling ability for handling sparse-view dynamic inputs. In parallel, we introduce a frequency-modulated module that regularizes the frequency spectrum of objects, thereby addressing the challenge of modeling sparse image inputs from a frequency-domain perspective. Experimental results demonstrate that our method achieves state-of-the-art view synthesis accuracy, object manipulation, and scene roaming ability in various scenes.