mlsanityData to 7 Oct 2026

Shape, Light, and Material Decomposition from Images using Monte Carlo Rendering and Denoising

arXiv 2206.03380 ↗First posted 7 Jun 2022cs.GR cs.CVcited by 163

Not on the map. Why?

Abstract (arXiv)

Recent advances in differentiable rendering have enabled high-quality reconstruction of 3D scenes from multi-view images. Most methods rely on simple rendering algorithms: pre-filtered direct lighting or learned representations of irradiance. We show that a more realistic shading model, incorporating ray tracing and Monte Carlo integration, substantially improves decomposition into shape, materials & lighting. Unfortunately, Monte Carlo integration provides estimates with significant noise, even at large sample counts, which makes gradient-based inverse rendering very challenging. To address this, we incorporate multiple importance sampling and denoising in a novel inverse rendering pipeline. This substantially improves convergence and enables gradient-based optimization at low sample counts. We present an efficient method to jointly reconstruct geometry (explicit triangle meshes), materials, and lighting, which substantially improves material and light separation compared to previous work. We argue that denoising can become an integral part of high quality inverse rendering pipelines.

Key facts. Cited by 163 papers in the map (data through 7 Oct 2026). Builds on 36 of its 83 references that are in the map. First posted on arXiv 7 Jun 2022.

Builds on

36 of 83 references.

References · 36

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
  4. 2021Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fieldscited by 1,957
  5. 2021NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstructioncited by 1,701
  6. 2020D-NeRF: Neural Radiance Fields for Dynamic Scenescited by 1,459
  7. 2020NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collectionscited by 1,230
  8. 2021PlenOctrees for Real-time Rendering of Neural Radiance Fieldscited by 904
  9. 2020NeRF++: Analyzing and Improving Neural Radiance Fieldscited by 857
  10. 2020GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fieldscited by 784
  11. 2019Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervisioncited by 699
  12. 2021KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPscited by 672
24 more
  1. 2021UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstructioncited by 643
  2. 2021FastNeRF: High-Fidelity Neural Rendering at 200FPScited by 510
  3. 2019BlendedMVS: A Large-scale Dataset for Generalized Multi-view Stereo Networkscited by 487
  4. 2021Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesiscited by 459
  5. 2020NeRD: Neural Reflectance Decomposition from Image Collectionscited by 446
  6. 2020Modular Primitives for High-Performance Differentiable Renderingcited by 439
  7. 2019Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoningcited by 420
  8. 2021Extracting Triangular 3D Models, Materials, and Lighting From Imagescited by 363
  9. 2021PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relightingcited by 332
  10. 2019Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderercited by 221
  11. 2021NeX: Real-time View Synthesis with Neural Basis Expansioncited by 215
  12. 2021Neural-PIL: Neural Pre-Integrated Lighting for Reflectance Decompositioncited by 204
  13. 2019Inverse Rendering for Complex Indoor Scenes: Shape, Spatially-Varying Lighting and SVBRDF from a Single Imagecited by 202
  14. 2020Neural Reflectance Fields for Appearance Acquisitioncited by 196
  15. 2022Modeling Indirect Illumination for Inverse Renderingcited by 167
  16. 2021NeRF--: Neural Radiance Fields Without Known Camera Parameterscited by 155
  17. 2020Image GANs meet Differentiable Rendering for Inverse Graphics and Interpretable 3D Neural Renderingcited by 101
  18. 2020Learning Deformable Tetrahedral Meshes for 3D Reconstructioncited by 84
  19. 2021Learning Indoor Inverse Rendering with 3D Spatially-Varying Lightingcited by 82
  20. 2021Unified Shape and SVBRDF Recovery using Differentiable Monte Carlo Renderingcited by 71
  21. 2020Two-shot Spatially-varying BRDF and Shape Estimationcited by 59
  22. 2021DIB-R++: Learning to Predict Lighting and Material with a Hybrid Differentiable Renderercited by 52
  23. 2021Appearance-Driven Automatic 3D Model Simplificationcited by 24
  24. 2022Towards Learning Neural Representations from Shadowscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 52; 2024: 54; 2025: 32; 2026: 22.

163 papers.

Sort:

By area:

  1. 2024SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusioncited by 364
  2. 2023GS-IR: 3D Gaussian Splatting for Inverse Renderingcited by 220
  3. 2023NeRO: Neural Geometry and BRDF Reconstruction of Reflective Objects from Multiview Imagescited by 163
  4. 2023Flexible Isosurface Extraction for Gradient-Based Mesh Optimizationcited by 138
  5. 2023Relightable 3D Gaussians: Realistic Point Cloud Relighting with BRDF Decomposition and Ray Tracingcited by 138
  6. 2023Delicate Textured Mesh Recovery from NeRF via Adaptive Surface Refinementcited by 106
  7. 20234K4D: Real-Time 4D View Synthesis at 4K Resolutioncited by 91
  8. 2024Neural Gaffer: Relighting Any Object via Diffusioncited by 91
  9. 2023Neural Fields meet Explicit Geometric Representation for Inverse Rendering of Urban Scenescited by 89
  10. 2023GIR: 3D Gaussian Inverse Rendering for Relightable Scene Factorizationcited by 86
  11. 2023BOP Challenge 2022 on Detection, Segmentation and Pose Estimation of Specific Rigid Objectscited by 70
  12. 2024IntrinsicAnything: Learning Diffusion Priors for Inverse Rendering Under Unknown Illuminationcited by 67
38 more
  1. 2024Meta 3D AssetGen: Text-to-Mesh Generation with High-Quality Geometry, Texture, and PBR Materialscited by 66
  2. 2024SF3D: Stable Fast 3D Mesh Reconstruction with UV-unwrapping and Illumination Disentanglementcited by 65
  3. 2023ENVIDR: Implicit Differentiable Renderer with Neural Environment Lightingcited by 63
  4. 2024BOP Challenge 2023 on Detection, Segmentation and Pose Estimation of Seen and Unseen Rigid Objectscited by 60
  5. 2024SCube: Instant Large-Scale Scene Reconstruction using VoxSplatscited by 56
  6. 2025DiffusionRenderer: Neural Inverse and Forward Rendering with Video Diffusion Modelscited by 55
  7. 2023Stanford-ORB: A Real-World 3D Object Inverse Rendering Benchmarkcited by 51
  8. 2024TextureDreamer: Image-guided Texture Synthesis through Geometry-aware Diffusioncited by 51
  9. 2024GoMAvatar: Efficient Animatable Human Modeling from Monocular Video Using Gaussians-on-Meshcited by 51
  10. 2024DreamMat: High-quality PBR Material Generation with Geometry- and Light-aware Diffusion Modelscited by 50
  11. 2023Neural-PBIR Reconstruction of Shape, Material, and Illuminationcited by 45
  12. 2023I2-SDF: Intrinsic Indoor Scene Reconstruction and Editing via Raytracing in Neural SDFscited by 42
  13. 2023Relighting Neural Radiance Fields with Shadow and Highlight Hintscited by 41
  14. 2025SPAR3D: Stable Point-Aware Reconstruction of 3D Objects from Single Imagescited by 41
  15. 2024DeferredGS: Decoupled and Editable Gaussian Splatting with Deferred Shadingcited by 40
  16. 2023NeFII: Inverse Rendering for Reflectance Decomposition with Near-Field Indirect Illuminationcited by 38
  17. 2025UniRelight: Learning Joint Decomposition and Synthesis for Video Relightingcited by 38
  18. 2023Neural Microfacet Fields for Inverse Renderingcited by 37
  19. 2024A Diffusion Approach to Radiance Field Relighting using Multi-Illumination Synthesiscited by 37
  20. 2024MeshAvatar: Learning High-quality Triangular Human Avatars from Multi-view Videoscited by 34
  21. 2023OpenIllumination: A Multi-Illumination Dataset for Inverse Rendering Evaluation on Real Objectscited by 31
  22. 2023Gaussian Splatting with NeRF-based Color and Opacitycited by 30
  23. 2023Nuvo: Neural UV Mapping for Unruly 3D Representationscited by 29
  24. 2024IllumiNeRF: 3D Relighting Without Inverse Renderingcited by 29
  25. 2025Digital Twin Catalog: A Large-Scale Photorealistic 3D Object Digital Twin Datasetcited by 29
  26. 2022HARP: Personalized Hand Reconstruction from a Monocular RGB Videocited by 28
  27. 2024MaterialFusion: Enhancing Inverse Rendering with Material Diffusion Priorscited by 28
  28. 2023VMesh: Hybrid Volume-Mesh Representation for Efficient View Synthesiscited by 25
  29. 2023NeISF: Neural Incident Stokes Field for Geometry and Material Estimationcited by 25
  30. 2023Ghost on the Shell: An Expressive Representation of General 3D Shapescited by 24
  31. 2023SEEAvatar: Photorealistic Text-to-3D Avatar Generation with Constrained Geometry and Appearancecited by 24
  32. 2024RelitLRM: Generative Relightable Radiance for Large Reconstruction Modelscited by 23
  33. 2023UrbanIR: Large-Scale Urban Scene Inverse Rendering from a Single Videocited by 22
  34. 2024GeoSplatting: Towards Geometry Guided Gaussian Splatting for Physically-based Inverse Renderingcited by 22
  35. 2025Relightable Full-Body Gaussian Codec Avatarscited by 22
  36. 2025SVG-IR: Spatially-Varying Gaussian Splatting for Inverse Renderingcited by 22
  37. 2022Latent Partition Implicit with Surface Codes for 3D Representationcited by 21
  38. 2023Dynamic Mesh-Aware Radiance Fieldscited by 21

50 most cited of 163 All 163 (JSON)