mlsanityData to 7 Oct 2026

Neural-PBIR Reconstruction of Shape, Material, and Illumination

arXiv 2304.13445 ↗First posted 26 Apr 2023cs.CVcited by 45

Not on the map. Why?

Abstract (arXiv)

Reconstructing the shape and spatially varying surface appearances of a physical-world object as well as its surrounding illumination based on 2D images (e.g., photographs) of the object has been a long-standing problem in computer vision and graphics. In this paper, we introduce an accurate and highly efficient object reconstruction pipeline combining neural based object reconstruction and physics-based inverse rendering (PBIR). Our pipeline firstly leverages a neural SDF based shape reconstruction to produce high-quality but potentially imperfect object shape. Then, we introduce a neural material and lighting distillation stage to achieve high-quality predictions for material and illumination. In the last stage, initialized by the neural predictions, we perform PBIR to refine the initial results and obtain the final high-quality reconstruction of object shape, material, and illumination. Experimental results demonstrate our pipeline significantly outperforms existing methods quality-wise and performance-wise.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). Builds on 29 of its 58 references that are in the map. First posted on arXiv 26 Apr 2023.

Builds on

29 of 58 references.

References · 29

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
  3. 2021NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstructioncited by 1,701
  4. 2021Plenoxels: Radiance Fields without Neural Networkscited by 1,607
  5. 2022TensoRF: Tensorial Radiance Fieldscited by 1,275
  6. 2021Volume Rendering of Neural Implicit Surfacescited by 990
  7. 2021Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstructioncited by 961
  8. 2021UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstructioncited by 643
  9. 2022MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstructioncited by 467
  10. 2020NeRD: Neural Reflectance Decomposition from Image Collectionscited by 446
  11. 2021Extracting Triangular 3D Models, Materials, and Lighting From Imagescited by 363
  12. 2021PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relightingcited by 332
17 more
  1. 2022NeuS2: Fast Learning of Neural Implicit Surfaces for Multi-view Reconstructioncited by 311
  2. 2022Geo-Neus: Geometry-Consistent Neural Implicit Surfaces Learning for Multi-view Reconstructioncited by 211
  3. 2021Neural-PIL: Neural Pre-Integrated Lighting for Reflectance Decompositioncited by 204
  4. 2022Modeling Indirect Illumination for Inverse Renderingcited by 167
  5. 2022Shape, Light, and Material Decomposition from Images using Monte Carlo Rendering and Denoisingcited by 163
  6. 2022Neural 3D Scene Reconstruction with the Manhattan-world Assumptioncited by 123
  7. 2019HorizonNet: Learning Room Layout with 1D Representation and Pano Stretch Data Augmentationcited by 107
  8. 2020HoHoNet: 360 Indoor Holistic Understanding with Latent Horizontal Featurescited by 104
  9. 2022Voxurf: Voxel-based Efficient and Accurate Neural Surface Reconstructioncited by 92
  10. 2021Unified Shape and SVBRDF Recovery using Differentiable Monte Carlo Renderingcited by 71
  11. 2019Inverse Path Tracing for Joint Material and Lighting Estimationcited by 62
  12. 2022Critical Regularizations for Neural Surface Reconstruction in the Wildcited by 51
  13. 2022MatteFormer: Transformer-Based Image Matting via Prior-Tokenscited by 48
  14. 2023NeFII: Inverse Rendering for Reflectance Decomposition with Near-Field Indirect Illuminationcited by 38
  15. 2022Differentiable Rendering of Neural SDFs through Reparameterizationcited by 26
  16. 2022Physics-Based Inverse Rendering using Combined Implicit and Explicit Geometriescited by 14
  17. 2021Indoor Panorama Planar 3D Reconstruction via Divide and Conquercited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 19; 2025: 15; 2026: 4.

45 papers.

Sort:

By area:

  1. 2023GIR: 3D Gaussian Inverse Rendering for Relightable Scene Factorizationcited by 86
  2. 2024IntrinsicAnything: Learning Diffusion Priors for Inverse Rendering Under Unknown Illuminationcited by 67
  3. 2023Stanford-ORB: A Real-World 3D Object Inverse Rendering Benchmarkcited by 51
  4. 2024TextureDreamer: Image-guided Texture Synthesis through Geometry-aware Diffusioncited by 51
  5. 2024DreamMat: High-quality PBR Material Generation with Geometry- and Light-aware Diffusion Modelscited by 50
  6. 2024Sparse Voxels Rasterization: Real-time High-fidelity Radiance Field Renderingcited by 35
  7. 2024IllumiNeRF: 3D Relighting Without Inverse Renderingcited by 29
  8. 2025Digital Twin Catalog: A Large-Scale Photorealistic 3D Object Digital Twin Datasetcited by 29
  9. 2024IDArb: Intrinsic Decomposition for Arbitrary Number of Input Views and Illuminationscited by 21
  10. 2024IRIS: Inverse Rendering of Indoor Scenes from Low Dynamic Range Imagescited by 19
  11. 2025LIRM: Large Inverse Rendering Model for Progressive Reconstruction of Shape, Materials and View-dependent Radiance Fieldscited by 16
  12. 2023Factored-NeuS: Reconstructing Surfaces, Illumination, and Materials of Possibly Glossy Objectscited by 15
33 more
  1. 2024Generative Multiview Relighting for 3D Reconstruction under Extreme Illumination Variationcited by 13
  2. 2025Gaussian Splatting with Discretized SDF for Relightable Assetscited by 12
  3. 2024GS-ROR2: Bidirectional-guided 3DGS and SDF for Reflective Object Relighting and Reconstructioncited by 10
  4. 2025Materialist: Physically Based Editing Using Single-Image Inverse Renderingcited by 10
  5. 2023Differentiable Point-based Inverse Renderingcited by 7
  6. 2024A Simple Approach to Differentiable Rendering of SDFscited by 7
  7. 2025ROGR: Relightable 3D Objects using Generative Relightingcited by 7
  8. 2023NeuS-PIR: Learning Relightable Neural Surface using Pre-Integrated Renderingcited by 6
  9. 2024Progressive Radiance Distillation for Inverse Rendering with Gaussian Splattingcited by 5
  10. 2024Uncertainty for SVBRDF Acquisition using Frequency Analysiscited by 4
  11. 2025DPCS: Path Tracing-Based Differentiable Projector-Camera Systemscited by 4
  12. 2024MAIR++: Improving Multi-view Attention Inverse Rendering with Implicit Lighting Representationcited by 3
  13. 2024PBIR-NIE: Glossy Object Capture under Non-Distant Lightingcited by 3
  14. 2025Differentiable Light Transport with Gaussian Surfels via Adapted Radiosity for Efficient Relighting and Geometry Reconstructioncited by 3
  15. 2023RNA: Relightable Neural Assetscited by 2
  16. 2024GaNI: Global and Near Field Illumination Aware Neural Inverse Renderingcited by 2
  17. 2024ESR-NeRF: Emissive Source Reconstruction Using LDR Multi-view Imagescited by 2
  18. 2024PIR: Photometric Inverse Rendering with Shading Cues Modeling and Surface Reflectance Regularizationcited by 2
  19. 2025ROSA: Reconstructing Object Shape and Appearance Textures by Adaptive Detail Transfercited by 2
  20. 2025Twinner: Shining Light on Digital Twins in a Few Snapscited by 2
  21. 2026LSRM: High-Fidelity Object-Centric Reconstruction via Scaled Context Windowscited by 2
  22. 2023SplitNeRF: Split Sum Approximation Neural Field for Joint Geometry, Illumination, and Material Estimationcited by 1
  23. 2024UniVoxel: Fast Inverse Rendering by Unified Voxelization of Scene Representationcited by 1
  24. 2024Physically-Based Photometric Bundle Adjustment in Non-Lambertian Environmentscited by 1
  25. 2025TexHOI: Reconstructing Textures of 3D Unknown Objects in Monocular Hand-Object Interaction Scenescited by 1
  26. 2025ReLumix: Extending Image Relighting to Video via Video Diffusion Modelscited by 1
  27. 2025GAINS: Gaussian-based Inverse Rendering from Sparse Multi-View Capturescited by 1
  28. 2025WildCap: Facial Albedo Capture in the Wild via Hybrid Inverse Renderingcited by 1
  29. 2025TranSplat: Instant Object Relighting in Gaussian Splatting via Spherical Harmonic Radiance Transfercited by 0
  30. 2025GLOW: Global Illumination-Aware Inverse Rendering of Indoor Scenes Captured with Dynamic Co-Located Light & Cameracited by 0
  31. 2026Diffusion-Based Material Regularization for Physics-Based Inverse Renderingcited by 0
  32. 2026RelightFormer: Feed-forward Generative Transformer for Multiview Object Relightingcited by 0
  33. 2026Integrating Multi-view Multi-light Surface Reconstruction into Cultural Heritage Workflowscited by 0