mlsanityData to 7 Oct 2026

OmniRe: Omni Urban Scene Reconstruction

arXiv 2408.16760 ↗First posted 29 Aug 2024cs.CVcited by 159

On the map: World models → World models for autonomous driving

Abstract (arXiv)

We introduce OmniRe, a comprehensive system for efficiently creating high-fidelity digital twins of dynamic real-world scenes from on-device logs. Recent methods using neural fields or Gaussian Splatting primarily focus on vehicles, hindering a holistic framework for all dynamic foregrounds demanded by downstream applications, e.g., the simulation of human behavior. OmniRe extends beyond vehicle modeling to enable accurate, full-length reconstruction of diverse dynamic objects in urban scenes. Our approach builds scene graphs on 3DGS and constructs multiple Gaussian representations in canonical spaces that model various dynamic actors, including vehicles, pedestrians, cyclists, and others. OmniRe allows holistically reconstructing any dynamic object in the scene, enabling advanced simulations (~60Hz) that include human-participated scenarios, such as pedestrian behavior simulation and human-vehicle interaction. This comprehensive simulation capability is unmatched by existing methods. Extensive evaluations on the Waymo dataset show that our approach outperforms prior state-of-the-art methods quantitatively and qualitatively by a large margin. We further extend our results to 5 additional popular driving datasets to demonstrate its generalizability on common urban scenes.

Key facts. Cited by 159 papers in the map (data through 7 Oct 2026). Placed in World models for autonomous driving, one of 11 directions in World models. Builds on 55 of its 63 references that are in the map. First posted on arXiv 29 Aug 2024.

Builds on

55 of 63 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Dynamic 3D Gaussian Fields for Urban Areas5 Jun
  2. 2021NuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles22 Jun

Other references · 53

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
  3. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  4. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
  5. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  6. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,436
  7. 2021Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fieldscited by 2,013
  8. 2021Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fieldscited by 1,957
  9. 2021Plenoxels: Radiance Fields without Neural Networkscited by 1,607
  10. 2020D-NeRF: Neural Radiance Fields for Dynamic Scenescited by 1,459
  11. 2020NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collectionscited by 1,230
  12. 2020Nerfies: Deformable Neural Radiance Fieldscited by 1,194
38 more
  1. 20234D Gaussian Splatting for Real-Time Dynamic Scene Renderingcited by 1,144
  2. 2023Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstructioncited by 753
  3. 2020Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenescited by 750
  4. 2023Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesiscited by 731
  5. 2022Block-NeRF: Scalable Large Scene Neural View Synthesiscited by 720
  6. 2023Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecastingcited by 701
  7. 2020Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Videocited by 470
  8. 2020Space-time Neural Irradiance Fields for Free-Viewpoint Videocited by 414
  9. 2023DrivingGaussian: Composite Gaussian Splatting for Surrounding Dynamic Autonomous Driving Scenescited by 359
  10. 2023Humans in 4D: Reconstructing and Tracking Humans with Transformerscited by 344
  11. 2023SC-GS: Sparse-Controlled Gaussian Splatting for Editable Dynamic Scenescited by 339
  12. 2018Learning 3D Human Dynamics from Videocited by 315
  13. 2020Neural Scene Graphs for Dynamic Scenescited by 298
  14. 2023UniSim: A Neural Closed-Loop Sensor Simulatorcited by 294
  15. 2021Urban Radiance Fieldscited by 263
  16. 2022NeuMan: Neural Human Radiance Field from a Single Videocited by 244
  17. 2022Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representationcited by 243
  18. 2021SNARF: Differentiable Forward Skinning for Animating Non-Rigid Neural Implicit Shapescited by 221
  19. 2023EmerNeRF: Emergent Spatial-Temporal Scene Decomposition via Self-Supervisioncited by 216
  20. 2021PandaSet: Advanced Sensor Suite Dataset for Autonomous Drivingcited by 186
  21. 2023MARS: An Instance-aware, Modular and Realistic Simulator for Autonomous Drivingcited by 170
  22. 2023GauHuman: Articulated Gaussian Splatting from Monocular Human Videoscited by 168
  23. 2020DeepCap: Monocular Human Performance Capture Using Weak Supervisioncited by 162
  24. 2021GLAMR: Global Occlusion-Aware Human Mesh Recovery with Dynamic Camerascited by 151
  25. 2023SUDS: Scalable Urban Dynamic Scenescited by 151
  26. 2023NeuRAD: Neural Rendering for Autonomous Drivingcited by 146
  27. 2023Periodic Vibration Gaussian: Dynamic Urban Scene Reconstruction and Real-time Renderingcited by 143
  28. 2024HUGS: Holistic Urban 3D Scene Understanding via Gaussian Splattingcited by 141
  29. 2022D2NeRF: Self-Supervised Decoupling of Dynamic and Static Objects from a Monocular Videocited by 136
  30. 2023GART: Gaussian Articulated Template Modelscited by 121
  31. 2023StreetSurf: Extending Multi-view Implicit Surface Reconstruction to Street Viewscited by 120
  32. 2018LiveCap: Real-time Human Performance Capture from Monocular Videocited by 93
  33. 2024Editable Scene Simulation for Autonomous Driving via Collaborative LLM-Agentscited by 91
  34. 2023MonoHuman: Animatable Human Neural Field from Monocular Videocited by 85
  35. 2024AbsGS: Recovering Fine Details for 3D Gaussian Splattingcited by 80
  36. 2022Neural Surface Reconstruction of Dynamic Scenes with Monocular RGB-D Cameracited by 57
  37. 2024AiOS: All-in-One-Stage Expressive Human Pose and Shape Estimationcited by 41
  38. 2024Multi-Level Neural Scene Graphs for Dynamic Urban Environmentscited by 27

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 21; 2025: 74; 2026: 64.

159 papers.

Sort:

By area:

  1. 2024DriveDreamer4D: World Models Are Effective Data Machines for 4D Driving Scene Representationcited by 123
  2. 2024ReconDreamer: Crafting World Models for Driving Scene Reconstruction via Online Restorationcited by 89
  3. 20243DGUT: Enabling Distorted Cameras and Secondary Rays in Gaussian Splattingcited by 87
  4. 2024HUGSIM: A Real-Time, Photo-Realistic and Closed-Loop Simulator for Autonomous Drivingcited by 84
  5. 2025Cosmos-Drive-Dreams: Scalable Synthetic Driving Data Generation with World Foundation Modelscited by 80
  6. 2024SplatAD: Real-Time Lidar and Camera Rendering with 3D Gaussian Splatting for Autonomous Drivingcited by 66
  7. 20253D and 4D World Modeling: A Surveycited by 64
  8. 2024InfiniCube: Unbounded and Controllable Dynamic 3D Driving Scene Generation with World-Guided Video Modelscited by 53
  9. 2024STORM: Spatio-Temporal Reconstruction Model for Large-Scale Outdoor Scenescited by 53
  10. 2024StreetCrafter: Street View Synthesis with Controllable Video Diffusion Modelscited by 49
  11. 2024Feed-Forward Bullet-Time Reconstruction of Dynamic Scenes from Monocular Videoscited by 48
  12. 2024DeSiRe-GS: 4D Street Gaussians for Static-Dynamic Decomposition and Surface Reconstruction for Urban Driving Scenescited by 45
38 more
  1. 2025DiST-4D: Disentangled Spatiotemporal Diffusion with Metric Depth for 4D Driving Scene Generationcited by 39
  2. 2026NeoVerse: Enhancing 4D World Model with in-the-wild Monocular Videoscited by 39
  3. 2025ReconDreamer++: Harmonizing Generative and Reconstructive Models for Driving Scene Representationcited by 36
  4. 2025SimScale: Learning to Drive via Real-World Simulation at Scalecited by 34
  5. 2024FreeSim: Toward Free-viewpoint Camera Simulation in Driving Scenescited by 30
  6. 2025GaussianDWM: 3D Gaussian Driving World Model for Unified Scene Understanding and Multi-Modal Generationcited by 23
  7. 2024Beyond Gaussians: Fast and High-Fidelity 3D Splatting with Linear Kernelscited by 22
  8. 2025MTGS: Multi-Traversal Gaussian Splattingcited by 17
  9. 2025AD-GS: Object-Aware B-Spline Gaussian Splatting for Self-Supervised Autonomous Drivingcited by 17
  10. 2025GSWorld: Closed-Loop Photo-Realistic Simulation Suite for Robotic Manipulationcited by 16
  11. 2024Bench2Drive-R: Turning Real World Data into Reactive Closed-Loop Autonomous Driving Benchmark by Generative Modelcited by 14
  12. 2025Unifying Appearance Codes and Bilateral Grids for Driving Scene Gaussian Splattingcited by 14
  13. 2025Rethinking Driving World Model as Synthetic Data Generator for Perception Taskscited by 14
  14. 2024EMD: Explicit Motion Modeling for High-Quality Street Gaussian Splattingcited by 13
  15. 2024Extrapolated Urban View Synthesis Benchmarkcited by 13
  16. 2025RoGSplat: Learning Robust Generalizable Human Gaussian Splatting from Sparse Multi-View Imagescited by 12
  17. 2025WeatherEdit: Controllable Weather Editing with 4D Gaussian Fieldcited by 11
  18. 2025Rig3R: Rig-Aware Conditioning for Learned 3D Reconstructioncited by 11
  19. 2025HAIF-GS: Hierarchical and Induced Flow-Guided Gaussian Splatting for Dynamic Scenecited by 11
  20. 2025GS-Occ3D: Scaling Vision-only Occupancy Reconstruction with Gaussian Splattingcited by 11
  21. 2025WorldSplat: Gaussian-Centric Feed-Forward 4D Scene Generation for Autonomous Drivingcited by 11
  22. 2024LiDAR-RT: Gaussian-based Ray Tracing for Dynamic LiDAR Re-simulationcited by 10
  23. 2025RealEngine: Simulating Autonomous Driving in Realistic Contextcited by 10
  24. 2026DiffusionHarmonizer: Bridging Neural Reconstruction and Photorealistic Simulation with Online Diffusion Enhancercited by 10
  25. 2025PINGS: Gaussian Splatting Meets Distance Fields within a Point-Based Implicit Neural Mapcited by 9
  26. 2025Learning-based 3D Reconstruction in Autonomous Driving: A Comprehensive Surveycited by 9
  27. 20253D Gaussian Inverse Rendering with Approximated Global Illuminationcited by 9
  28. 2025VoxelSplat: Dynamic Gaussian Splatting as an Effective Loss for Occupancy and Flow Predictioncited by 9
  29. 2024LiHi-GS: LiDAR-Supervised Gaussian Splatting for Highway Driving Scene Reconstructioncited by 8
  30. 20254D Driving Scene Generation With Stereo Forcingcited by 8
  31. 2024UrbanGS: Semantic-Guided Gaussian Splatting for Urban Scene Reconstructioncited by 7
  32. 2025Generative AI for Autonomous Driving: Frontiers and Opportunitiescited by 7
  33. 2025Flux4D: Flow-based Unsupervised 4D Reconstructioncited by 7
  34. 2026ParkGaussian: Surround-view 3D Gaussian Splatting for Autonomous Parkingcited by 7
  35. 2025Realistic and Controllable 3D Gaussian-Guided Object Editing for Driving Video Generationcited by 6
  36. 2025OpenBox: Annotate Any Bounding Boxes in 3Dcited by 6
  37. 2025B'ezierGS: Dynamic Urban Scene Reconstruction with B'ezier Curve Gaussian Splattingcited by 5
  38. 2025UrbanVerse: Scaling Urban Simulation by Watching City-Tour Videoscited by 5

50 most cited of 159 All 159 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Street Gaussians: Modeling Dynamic Urban Scenes with Gaussian Splatting
  2. 2024DynamicCity: Large-Scale 4D Occupancy Generation from Dynamic Scenes
  3. 2025VDEGaussian: Video Diffusion Enhanced 4D Gaussian Splatting for Dynamic Urban Scenes Modeling
  4. 2026NVIDIA OmniDreams: Real-Time Generative World Model for Closed-Loop Autonomous Vehicle Simulation
  5. 2026Mirage2Matter: A Physically Grounded Gaussian World Model from Video
  6. 2026OmniX: Any-view and Any-time 4D Reconstruction via Feed-forward Trajectory Fields
  7. 2026SpectralSplat: Appearance-Disentangled Feed-Forward Gaussian Splatting for Driving Scenes
  8. 2025OmnimatteZero: Fast Training-free Omnimatte with Pre-trained Video Diffusion Models