mlsanityData to 7 Oct 2026

Neural Fields meet Explicit Geometric Representation for Inverse Rendering of Urban Scenes

arXiv 2304.03266 ↗First posted 6 Apr 2023cs.CVcited by 89

Not on the map. Why?

Abstract (arXiv)

Reconstruction and intrinsic decomposition of scenes from captured imagery would enable many applications such as relighting and virtual object insertion. Recent NeRF based methods achieve impressive fidelity of 3D reconstruction, but bake the lighting and shadows into the radiance field, while mesh-based methods that facilitate intrinsic decomposition through differentiable rendering have not yet scaled to the complexity and scale of outdoor scenes. We present a novel inverse rendering framework for large urban scenes capable of jointly reconstructing the scene geometry, spatially-varying materials, and HDR lighting from a set of posed RGB images with optional depth. Specifically, we use a neural field to account for the primary rays, and use an explicit mesh (reconstructed from the underlying neural field) for modeling secondary rays that produce higher-order lighting effects such as cast shadows. By faithfully disentangling complex geometry and materials from lighting effects, our method enables photorealistic relighting with specular and shadow effects on several outdoor datasets. Moreover, it supports physics-based scene manipulations such as virtual object insertion with ray-traced shadow casting.

Key facts. Cited by 89 papers in the map (data through 7 Oct 2026). Builds on 52 of its 74 references that are in the map. First posted on arXiv 6 Apr 2023.

Builds on

52 of 74 references.

References · 52

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
  4. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,436
  5. 2021NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstructioncited by 1,701
  6. 2021Plenoxels: Radiance Fields without Neural Networkscited by 1,607
  7. 2020NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collectionscited by 1,230
  8. 2021Volume Rendering of Neural Implicit Surfacescited by 990
  9. 2021Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstructioncited by 961
  10. 2020NeRF++: Analyzing and Improving Neural Radiance Fieldscited by 857
  11. 2022Block-NeRF: Scalable Large Scene Neural View Synthesiscited by 720
  12. 2021Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fieldscited by 569
38 more
  1. 2020NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesiscited by 493
  2. 2020NeRD: Neural Reflectance Decomposition from Image Collectionscited by 446
  3. 2021Mega-NeRF: Scalable Construction of Large-Scale NeRFs for Virtual Fly-Throughscited by 396
  4. 2021Extracting Triangular 3D Models, Materials, and Lighting From Imagescited by 363
  5. 2021PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relightingcited by 332
  6. 2021Urban Radiance Fieldscited by 263
  7. 2021BungeeNeRF: Progressive Neural Radiance Field for Extreme Multi-scale Scene Renderingcited by 228
  8. 2019Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderercited by 221
  9. 2021Neural-PIL: Neural Pre-Integrated Lighting for Reflectance Decompositioncited by 204
  10. 2019Inverse Rendering for Complex Indoor Scenes: Shape, Spatially-Varying Lighting and SVBRDF from a Single Imagecited by 202
  11. 2022Monocular Dynamic View Synthesis: A Reality Checkcited by 199
  12. 2020Neural Reflectance Fields for Appearance Acquisitioncited by 196
  13. 2020Hierarchical Multi-Scale Attention for Semantic Segmentationcited by 183
  14. 2022Modeling Indirect Illumination for Inverse Renderingcited by 167
  15. 2022Shape, Light, and Material Decomposition from Images using Monte Carlo Rendering and Denoisingcited by 163
  16. 2021NeRS: Neural Reflectance Surfaces for Sparse-view 3D Reconstruction in the Wildcited by 136
  17. 2021NeRFactor: Neural Factorization of Shape and Reflectance Under an Unknown Illuminationcited by 135
  18. 2021NeRF for Outdoor Scene Relightingcited by 114
  19. 2022IRON: Inverse Rendering by Optimizing Neural SDFs and Materials from Photometric Imagescited by 110
  20. 2022NeILF: Neural Incident Light Field for Physically-based Material Estimationcited by 105
  21. 2019Neural Inverse Rendering of an Indoor Scene from a Single Imagecited by 101
  22. 2018InverseRenderNet: Learning single image inverse renderingcited by 100
  23. 2020Deep Reflectance Volumes: Relightable Reconstructions from Multi-View Photometric Imagescited by 98
  24. 2022SAMURAI: Shape And Material from Unconstrained Real-world Arbitrary Image collectionscited by 89
  25. 2018CGIntrinsics: Better Intrinsic Image Decomposition through Physically-Based Renderingcited by 85
  26. 2021Learning Indoor Inverse Rendering with 3D Spatially-Varying Lightingcited by 82
  27. 2019Fast Spatially-Varying Indoor Lighting Estimationcited by 72
  28. 2019Deep Sky Modeling for Single Image Outdoor Lighting Estimationcited by 66
  29. 2019DeepLight: Learning Illumination for Unconstrained Mobile Mixed Realitycited by 63
  30. 2020Two-shot Spatially-varying BRDF and Shape Estimationcited by 59
  31. 2016Deep Outdoor Illumination Estimationcited by 53
  32. 2021DIB-R++: Learning to Predict Lighting and Material with a Hybrid Differentiable Renderercited by 52
  33. 2019All-Weather Deep Outdoor Lighting Estimationcited by 41
  34. 2020Learning to Factorize and Relight a Citycited by 36
  35. 2022NeROIC: Neural Rendering of Objects from Online Image Collectionscited by 36
  36. 2022Neural Light Field Estimation for Street Scenes with Differentiable Virtual Object Insertioncited by 35
  37. 2022De-rendering 3D Objects in the Wildcited by 31
  38. 2017Shading Annotations in the Wildcited by 19

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 15; 2024: 46; 2025: 22; 2026: 6.

89 papers.

Sort:

By area:

  1. 2023DrivingGaussian: Composite Gaussian Splatting for Surrounding Dynamic Autonomous Driving Scenescited by 359
  2. 2024Street Gaussians: Modeling Dynamic Urban Scenes with Gaussian Splattingcited by 218
  3. 2023EmerNeRF: Emergent Spatial-Temporal Scene Decomposition via Self-Supervisioncited by 216
  4. 2024RGBleftrightarrowX: Image decomposition and synthesis using material- and lighting-aware diffusion modelscited by 159
  5. 2023Periodic Vibration Gaussian: Dynamic Urban Scene Reconstruction and Real-time Renderingcited by 143
  6. 2023StreetSurf: Extending Multi-view Implicit Surface Reconstruction to Street Viewscited by 120
  7. 2025Cosmos-Drive-Dreams: Scalable Synthetic Driving Data Generation with World Foundation Modelscited by 80
  8. 2024LoopSplat: Loop Closure by Registering 3D Gaussian Splatscited by 59
  9. 2024LightIt: Illumination Modeling and Control for Diffusion Modelscited by 56
  10. 2025DiffusionRenderer: Neural Inverse and Forward Rendering with Video Diffusion Modelscited by 55
  11. 2024FreeVS: Generative View Synthesis on Free Driving Trajectorycited by 52
  12. 2024DeferredGS: Decoupled and Editable Gaussian Splatting with Deferred Shadingcited by 40
38 more
  1. 2025UniRelight: Learning Joint Decomposition and Synthesis for Video Relightingcited by 38
  2. 2024A Diffusion Approach to Radiance Field Relighting using Multi-Illumination Synthesiscited by 37
  3. 2024Dynamic 3D Gaussian Fields for Urban Areascited by 35
  4. 2024Gaussian Splashing: Unified Particles for Versatile Motion Synthesis and Renderingcited by 34
  5. 2024DistillNeRF: Perceiving 3D Scenes from Single-Glance Images by Distilling Neural Fields and Foundation Model Featurescited by 33
  6. 2023SpecNeRF: Gaussian Directional Encoding for Specular Reflectionscited by 30
  7. 2024Neural Material Adaptor for Visual Grounding of Intrinsic Dynamicscited by 30
  8. 2024G3R: Gradient Guided Generalizable Reconstructioncited by 29
  9. 2024S-NeRF++: Autonomous Driving Simulation via Neural Reconstruction and Generationcited by 28
  10. 2024Multi-Level Neural Scene Graphs for Dynamic Urban Environmentscited by 27
  11. 2024Photorealistic Object Insertion with Diffusion-Guided Inverse Renderingcited by 27
  12. 2024LiDAR4D: Dynamic Neural Fields for Novel Space-time View LiDAR Synthesiscited by 26
  13. 2023NeISF: Neural Incident Stokes Field for Geometry and Material Estimationcited by 25
  14. 2024NeRF-HuGS: Improved Neural Radiance Fields in Non-static Scenes Using Heuristics-Guided Segmentationcited by 25
  15. 2023UC-NeRF: Neural Radiance Field for Under-Calibrated Multi-view Cameras in Autonomous Drivingcited by 23
  16. 2023UrbanIR: Large-Scale Urban Scene Inverse Rendering from a Single Videocited by 22
  17. 2023Dynamic LiDAR Re-simulation using Compositional Neural Fieldscited by 22
  18. 2024LumiGauss: Relightable Gaussian Splatting in the Wildcited by 21
  19. 2023LightSim: Neural Lighting Simulation for Urban Scenescited by 20
  20. 2024IRIS: Inverse Rendering of Indoor Scenes from Low Dynamic Range Imagescited by 19
  21. 2024Neural Radiance Field in Autonomous Driving: A Surveycited by 19
  22. 2024Radar Fields: Frequency-Space Neural Scene Representations for FMCW Radarcited by 18
  23. 2024Multiagent Multitraversal Multimodal Self-Driving: Open MARS Datasetcited by 18
  24. 2024NeRFs in Robotics: A Surveycited by 16
  25. 2025Ouroboros: Single-step Diffusion Models for Cycle-consistent Forward and Inverse Renderingcited by 16
  26. 2023Applications of Large Scale Foundation Models for Autonomous Drivingcited by 14
  27. 2024Gaussian Shadow Casting for Neural Characterscited by 14
  28. 2024LidaRF: Delving into Lidar for Neural Radiance Field on Street Scenescited by 14
  29. 2025LuxDiT: Lighting Estimation with Video Diffusion Transformercited by 14
  30. 2024Fed3DGS: Scalable 3D Gaussian Splatting with Federated Learningcited by 12
  31. 2024PBR-NeRF: Inverse Rendering with Physics-Based Neural Fieldscited by 12
  32. 2025RainyGS: Efficient Rain Synthesis with Physically-Based Gaussian Splattingcited by 12
  33. 2023Digging into Depth Priors for Outdoor Neural Radiance Fieldscited by 11
  34. 2023The Sky's the Limit: Re-lightable Outdoor Scenes via a Sky-pixel Constrained Illumination Prior and Outside-In Visibilitycited by 11
  35. 2024Driving View Synthesis on Free-form Trajectories with Generative Priorcited by 11
  36. 2025GaSLight: Gaussian Splats for Spatially-Varying Lighting in HDRcited by 11
  37. 2025RealEngine: Simulating Autonomous Driving in Realistic Contextcited by 10
  38. 2025Learning-based 3D Reconstruction in Autonomous Driving: A Comprehensive Surveycited by 9

50 most cited of 89 All 89 (JSON)