mlsanityData to 7 Oct 2026

Incorporating dense metric depth into neural 3D representations for view synthesis and relighting

arXiv 2409.03061 ↗First posted 4 Sep 2024cs.CV cs.GR cs.ROcited by 0

Not on the map. Why?

Abstract (arXiv)

Synthesizing accurate geometry and photo-realistic appearance of small scenes is an active area of research with compelling use cases in gaming, virtual reality, robotic-manipulation, autonomous driving, convenient product capture, and consumer-level photography. When applying scene geometry and appearance estimation techniques to robotics, we found that the narrow cone of possible viewpoints due to the limited range of robot motion and scene clutter caused current estimation techniques to produce poor quality estimates or even fail. On the other hand, in robotic applications, dense metric depth can often be measured directly using stereo and illumination can be controlled. Depth can provide a good initial estimate of the object geometry to improve reconstruction, while multi-illumination images can facilitate relighting. In this work we demonstrate a method to incorporate dense metric depth into the training of neural 3D representations and address an artifact observed while jointly refining geometry and appearance by disambiguating between texture and geometry edges. We also discuss a multi-flash stereo camera system developed to capture the necessary data for our pipeline and show results on relighting and view synthesis with a few training views.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 57 of its 116 references that are in the map. First posted on arXiv 4 Sep 2024.

Builds on

57 of 116 references.

References · 57

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
  3. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
  4. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,170
  5. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,037
  6. 2021NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstructioncited by 1,701
  7. 2021Plenoxels: Radiance Fields without Neural Networkscited by 1,607
  8. 2020NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collectionscited by 1,230
  9. 2021Volume Rendering of Neural Implicit Surfacescited by 990
  10. 2019The Replica Dataset: A Digital Replica of Indoor Spacescited by 885
  11. 2021Depth-supervised NeRF: Fewer Views and Faster Training for Freecited by 732
  12. 2020Implicit Geometric Regularization for Learning Shapescited by 707
38 more
  1. 2021UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstructioncited by 643
  2. 2023Nerfstudio: A Modular Framework for Neural Radiance Field Developmentcited by 596
  3. 2018From Coarse to Fine: Robust Hierarchical Localization at Large Scalecited by 556
  4. 2019BlendedMVS: A Large-scale Dataset for Generalized Multi-view Stereo Networkscited by 487
  5. 2022MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstructioncited by 467
  6. 2023Neuralangelo: High-Fidelity Neural Surface Reconstructioncited by 464
  7. 2021Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets from 3D Scanscited by 336
  8. 2021NeRF in the Dark: High Dynamic Range View Synthesis from Noisy Raw Imagescited by 336
  9. 2021GMFlow: Learning Optical Flow via Global Matchingcited by 324
  10. 2021Neural RGB-D Surface Reconstructioncited by 312
  11. 2021Dense Depth Priors for Neural Radiance Fields from Sparse Input Viewscited by 301
  12. 2023BakedSDF: Meshing Neural SDFs for Real-Time View Synthesiscited by 220
  13. 2023MERF: Memory-Efficient Radiance Fields for Real-time View Synthesis in Unbounded Scenescited by 215
  14. 2022Geo-Neus: Geometry-Consistent Neural Implicit Surfaces Learning for Multi-view Reconstructioncited by 211
  15. 2020Neural Reflectance Fields for Appearance Acquisitioncited by 196
  16. 2023NeRO: Neural Geometry and BRDF Reconstruction of Reflective Objects from Multiview Imagescited by 163
  17. 2023Point-SLAM: Dense Neural Point Cloud-based SLAMcited by 160
  18. 2018Learning Depth with Convolutional Spatial Propagation Networkcited by 135
  19. 2016BundleFusion: Real-time Globally Consistent 3D Reconstruction using On-the-fly Surface Re-integrationcited by 116
  20. 2022IRON: Inverse Rendering by Optimizing Neural SDFs and Materials from Photometric Imagescited by 110
  21. 2023Bayes' Rays: Uncertainty Quantification for Neural Radiance Fieldscited by 90
  22. 2021T"oRF: Time-of-Flight Radiance Fields for Dynamic Scene View Synthesiscited by 88
  23. 2017Intel RealSense Stereoscopic Depth Camerascited by 82
  24. 2022Neural 3D Reconstruction in the Wildcited by 79
  25. 2022NerfAcc: A General NeRF Acceleration Toolboxcited by 73
  26. 2022E-NeRF: Neural Radiance Fields from a Moving Event Cameracited by 70
  27. 2023Adaptive Shells for Efficient Neural Radiance Field Renderingcited by 69
  28. 2023SMERF: Streamable Memory Efficient Radiance Fields for Real-Time Large-Scene Explorationcited by 62
  29. 2020Multi-View Photometric Stereo: A Robust Solution and Benchmark Dataset for Spatially Varying Isotropic Materialscited by 56
  30. 2022PANDORA: Polarization-Aided Neural Decomposition Of Radiancecited by 52
  31. 2019Self-calibrating Deep Photometric Stereo Networkscited by 47
  32. 2023Robust e-NeRF: NeRF from Sparse & Noisy Events under Non-Uniform Motioncited by 44
  33. 2023ReLight My NeRF: A Dataset for Novel View Synthesis and Relighting of Real World Objectscited by 42
  34. 2022FWD: Real-time Novel View Synthesis with Forward Warping and Depthcited by 39
  35. 2023OpenIllumination: A Multi-Illumination Dataset for Inverse Rendering Evaluation on Real Objectscited by 31
  36. 2023Nuvo: Neural UV Mapping for Unruly 3D Representationscited by 29
  37. 2022PIE-Net: Photometric Invariant Edge Guided Network for Intrinsic Image Decompositioncited by 26
  38. 2020Du2Net: Learning Depth Estimation from Dual-Cameras and Dual-Pixelscited by 23

50 most cited of 57 All 57 (JSON)

Built on by

Not cited yet.