mlsanityData to 14 Oct 2026

PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relighting

arXiv 2104.00674 ↗First posted 1 Apr 2021cs.CV cs.GRcited by 335

Not on the map. Why?

Abstract (arXiv)

We present PhySG, an end-to-end inverse rendering pipeline that includes a fully differentiable renderer and can reconstruct geometry, materials, and illumination from scratch from a set of RGB input images. Our framework represents specular BRDFs and environmental illumination using mixtures of spherical Gaussians, and represents geometry as a signed distance function parameterized as a Multi-Layer Perceptron. The use of spherical Gaussians allows us to efficiently solve for approximate light transport, and our method works on scenes with challenging non-Lambertian reflectance captured under natural, static illumination. We demonstrate, with both synthetic and real data, that our reconstructions not only enable rendering of novel viewpoints, but also physics-based appearance editing of materials and illumination.

Key facts. Cited by 335 papers in the map (data through 14 Oct 2026). Builds on 25 of its 56 references that are in the map. First posted on arXiv 1 Apr 2021.

Builds on

25 of 56 references.

References · 25

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  3. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,689
  4. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,200
  5. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,041
  6. 2020NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collectionscited by 1,232
  7. 2019Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representationscited by 864
  8. 2020NeRF++: Analyzing and Improving Neural Radiance Fieldscited by 861
  9. 2020Implicit Geometric Regularization for Learning Shapescited by 710
  10. 2019Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervisioncited by 700
  11. 2018DeepVoxels: Learning Persistent 3D Feature Embeddingscited by 405
  12. 2020State of the Art on Neural Renderingcited by 278
13 more
  1. 2019Deferred Neural Rendering: Image Synthesis using Neural Texturescited by 253
  2. 2019Inverse Rendering for Complex Indoor Scenes: Shape, Spatially-Varying Lighting and SVBRDF from a Single Imagecited by 203
  3. 2020Neural Reflectance Fields for Appearance Acquisitioncited by 196
  4. 2019SDFDiff: Differentiable Rendering of Signed Distance Fields for 3D Shape Optimizationcited by 159
  5. 2019Neural Rerendering in the Wildcited by 121
  6. 2020Deep Reflectance Volumes: Relightable Reconstructions from Multi-View Photometric Imagescited by 98
  7. 2020Deep 3D Capture: Geometry and Reflectance from Sparse Multi-View Imagescited by 65
  8. 2019Inverse Path Tracing for Joint Material and Lighting Estimationcited by 62
  9. 2020Crowdsampling the Plenoptic Functioncited by 62
  10. 2017Intrinsic3D: High-Quality 3D Reconstruction by Joint Appearance and Geometry Optimization with Spatially-Varying Lightingcited by 31
  11. 2020Seeing the World in a Bag of Chipscited by 25
  12. 2020Learning Implicit Surface Light Fieldscited by 23
  13. 2016Radiometric Scene Decomposition: Scene Reflectance, Illumination, and Geometry from RGB-D Imagescited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 10; 2022: 53; 2023: 100; 2024: 86; 2025: 50; 2026: 36.

335 papers.

Sort:

By area:

  1. 2021NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstructioncited by 1,712
  2. 2023SuGaR: Surface-Aligned Gaussian Splatting for Efficient 3D Mesh Reconstruction and High-Quality Mesh Renderingcited by 616
  3. 2021Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fieldscited by 575
  4. 2021Neural Fields in Visual Computing and Beyondcited by 540
  5. 2022GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Imagescited by 446
  6. 2021Extracting Triangular 3D Models, Materials, and Lighting From Imagescited by 366
  7. 2021Advances in Neural Renderingcited by 365
  8. 2024SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusioncited by 364
  9. 2023GS-IR: 3D Gaussian Splatting for Inverse Renderingcited by 224
  10. 2023BakedSDF: Meshing Neural SDFs for Real-Time View Synthesiscited by 221
  11. 2021Neural-PIL: Neural Pre-Integrated Lighting for Reflectance Decompositioncited by 205
  12. 2022Modeling Indirect Illumination for Inverse Renderingcited by 168
38 more
  1. 2023TensoIR: Tensorial Inverse Renderingcited by 168
  2. 2022Shape, Light, and Material Decomposition from Images using Monte Carlo Rendering and Denoisingcited by 163
  3. 2023NeRO: Neural Geometry and BRDF Reconstruction of Reflective Objects from Multiview Imagescited by 163
  4. 2024High-quality Surface Reconstruction using Gaussian Surfelscited by 161
  5. 2023Relightable 3D Gaussians: Realistic Point Cloud Relighting with BRDF Decomposition and Ray Tracingcited by 142
  6. 2021NeRFactor: Neural Factorization of Shape and Reflectance Under an Unknown Illuminationcited by 135
  7. 2023Relightable Gaussian Codec Avatarscited by 127
  8. 2023F2-NeRF: Fast Neural Radiance Field Training with Free Camera Trajectoriescited by 118
  9. 2022IRON: Inverse Rendering by Optimizing Neural SDFs and Materials from Photometric Imagescited by 111
  10. 2023Delicate Textured Mesh Recovery from NeRF via Adaptive Surface Refinementcited by 106
  11. 2022NeILF: Neural Incident Light Field for Physically-based Material Estimationcited by 105
  12. 2024Neural Gaffer: Relighting Any Object via Diffusioncited by 91
  13. 2023Neural Fields meet Explicit Geometric Representation for Inverse Rendering of Urban Scenescited by 90
  14. 2022SAMURAI: Shape And Material from Unconstrained Real-world Arbitrary Image collectionscited by 89
  15. 2023GIR: 3D Gaussian Inverse Rendering for Relightable Scene Factorizationcited by 87
  16. 2022TANGO: Text-driven Photorealistic and Robust 3D Stylization via Lighting Decompositioncited by 83
  17. 2024Fully Explicit Dynamic Gaussian Splattingcited by 76
  18. 2022NerfAcc: A General NeRF Acceleration Toolboxcited by 74
  19. 2023NeILF++: Inter-Reflectable Light Fields for Geometry and Material Estimationcited by 74
  20. 2022PS-NeRF: Neural Inverse Rendering for Multi-view Photometric Stereocited by 69
  21. 2023Ref-NeuS: Ambiguity-Reduced Neural Implicit Surface Learning for Multi-View Reconstruction with Reflectioncited by 69
  22. 2023Adaptive Shells for Efficient Neural Radiance Field Renderingcited by 69
  23. 2024IntrinsicAnything: Learning Diffusion Priors for Inverse Rendering Under Unknown Illuminationcited by 67
  24. 2022LASSIE: Learning Articulated Shapes from Sparse Image Ensemble via 3D Part Discoverycited by 66
  25. 2024Meta 3D AssetGen: Text-to-Mesh Generation with High-Quality Geometry, Texture, and PBR Materialscited by 66
  26. 2023ENVIDR: Implicit Differentiable Renderer with Neural Environment Lightingcited by 65
  27. 2024SF3D: Stable Fast 3D Mesh Reconstruction with UV-unwrapping and Illumination Disentanglementcited by 65
  28. 2024Advances in 3D Generation: A Surveycited by 63
  29. 20243D Gaussian Splatting with Deferred Reflectioncited by 63
  30. 2023Factor Fields: A Unified Framework for Neural Fields and Beyondcited by 58
  31. 2025DiffusionRenderer: Neural Inverse and Forward Rendering with Video Diffusion Modelscited by 57
  32. 2022Parallel Inversion of Neural Radiance Fields for Robust Pose Estimationcited by 55
  33. 2021DIB-R++: Learning to Predict Lighting and Material with a Hybrid Differentiable Renderercited by 52
  34. 2022PANDORA: Polarization-Aided Neural Decomposition Of Radiancecited by 52
  35. 2023Stanford-ORB: A Real-World 3D Object Inverse Rendering Benchmarkcited by 51
  36. 2022IntrinsicNeRF: Learning Intrinsic Neural Radiance Fields for Editable Novel View Synthesiscited by 50
  37. 2024DreamMat: High-quality PBR Material Generation with Geometry- and Light-aware Diffusion Modelscited by 50
  38. 2022PaletteNeRF: Palette-based Appearance Editing of Neural Radiance Fieldscited by 45

50 most cited of 335 All 335 (JSON)