mlsanityData to 7 Oct 2026

Extracting Triangular 3D Models, Materials, and Lighting From Images

arXiv 2111.12503 ↗First posted 24 Nov 2021cs.CV cs.GRcited by 363

Not on the map. Why?

Cited by 37 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present an efficient method for joint optimization of topology, materials and lighting from multi-view image observations. Unlike recent multi-view reconstruction approaches, which typically produce entangled 3D representations encoded in neural networks, we output triangle meshes with spatially-varying materials and environment lighting that can be deployed in any traditional graphics engine unmodified. We leverage recent work in differentiable rendering, coordinate-based networks to compactly represent volumetric texturing, alongside differentiable marching tetrahedrons to enable gradient-based optimization directly on the surface mesh. Finally, we introduce a differentiable formulation of the split sum approximation of environment lighting to efficiently recover all-frequency lighting. Experiments show our extracted models used in advanced scene editing, material decomposition, and high quality view interpolation, all running at interactive rates in triangle-based renderers (rasterizers and path tracers). Project website: https://nvlabs.github.io/nvdiffrec/ .

Key facts. Cited by 363 papers in the map (data through 7 Oct 2026). Builds on 33 of its 81 references that are in the map. First posted on arXiv 24 Nov 2021.

Builds on

33 of 81 references.

References · 33

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2021NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstructioncited by 1,701
  3. 2020D-NeRF: Neural Radiance Fields for Dynamic Scenescited by 1,459
  4. 2020NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collectionscited by 1,230
  5. 2021PlenOctrees for Real-time Rendering of Neural Radiance Fieldscited by 904
  6. 2020NeRF++: Analyzing and Improving Neural Radiance Fieldscited by 857
  7. 2020GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fieldscited by 784
  8. 2019Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervisioncited by 699
  9. 2021KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPscited by 672
  10. 2021UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstructioncited by 643
  11. 2021FastNeRF: High-Fidelity Neural Rendering at 200FPScited by 510
  12. 2020NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesiscited by 493
21 more
  1. 2021Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesiscited by 459
  2. 2020NeRD: Neural Reflectance Decomposition from Image Collectionscited by 446
  3. 2019PointRend: Image Segmentation as Renderingcited by 439
  4. 2020Modular Primitives for High-Performance Differentiable Renderingcited by 439
  5. 2019Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoningcited by 420
  6. 2021PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relightingcited by 332
  7. 2021Learning Object-Compositional Neural Radiance Field for Editable Scene Renderingcited by 228
  8. 2019Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderercited by 221
  9. 2021NeX: Real-time View Synthesis with Neural Basis Expansioncited by 215
  10. 2021Neural-PIL: Neural Pre-Integrated Lighting for Reflectance Decompositioncited by 204
  11. 2019Inverse Rendering for Complex Indoor Scenes: Shape, Spatially-Varying Lighting and SVBRDF from a Single Imagecited by 202
  12. 2020Neural Reflectance Fields for Appearance Acquisitioncited by 196
  13. 2019SDFDiff: Differentiable Rendering of Signed Distance Fields for 3D Shape Optimizationcited by 159
  14. 2020Image GANs meet Differentiable Rendering for Inverse Graphics and Interpretable 3D Neural Renderingcited by 101
  15. 2019Kaolin: A PyTorch Library for Accelerating 3D Deep Learning Researchcited by 98
  16. 2020Learning Deformable Tetrahedral Meshes for 3D Reconstructioncited by 84
  17. 2021Unified Shape and SVBRDF Recovery using Differentiable Monte Carlo Renderingcited by 71
  18. 2020Two-shot Spatially-varying BRDF and Shape Estimationcited by 59
  19. 2021SAPE: Spatially-Adaptive Progressive Encoding for Neural Optimizationcited by 59
  20. 2021DIB-R++: Learning to Predict Lighting and Material with a Hybrid Differentiable Renderercited by 52
  21. 2021Appearance-Driven Automatic 3D Model Simplificationcited by 24

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 29; 2023: 125; 2024: 105; 2025: 62; 2026: 42.

363 papers.

Sort:

By area:

  1. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,285
  2. 2023Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creationcited by 684
  3. 2023Scaffold-GS: Structured 3D Gaussians for View-Adaptive Renderingcited by 684
  4. 2023Instruct-NeRF2NeRF: Editing 3D Scenes with Instructionscited by 482
  5. 2022GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Imagescited by 446
  6. 2022MobileNeRF: Exploiting the Polygon Rasterization Pipeline for Efficient Neural Field Rendering on Mobile Architecturescited by 308
  7. 2023GS-IR: 3D Gaussian Splatting for Inverse Renderingcited by 220
  8. 2023MERF: Memory-Efficient Radiance Fields for Real-time View Synthesis in Unbounded Scenescited by 215
  9. 2023MeshDiffusion: Score-based Generative 3D Mesh Modelingcited by 198
  10. 2023GPS-Gaussian: Generalizable Pixel-wise 3D Gaussian Splatting for Real-time Human Novel View Synthesiscited by 170
  11. 2023RichDreamer: A Generalizable Normal-Depth Diffusion Model for Detail Richness in Text-to-3Dcited by 168
  12. 2022Shape, Light, and Material Decomposition from Images using Monte Carlo Rendering and Denoisingcited by 163
38 more
  1. 2023NeRO: Neural Geometry and BRDF Reconstruction of Reflective Objects from Multiview Imagescited by 163
  2. 2025Native and Compact Structured Latents for 3D Generationcited by 157
  3. 2023DreamAvatar: Text-and-Shape Guided 3D Human Avatar Generation via Diffusion Modelscited by 154
  4. 2023BundleSDF: Neural 6-DoF Tracking and 3D Reconstruction of Unknown Objectscited by 144
  5. 2023Flexible Isosurface Extraction for Gradient-Based Mesh Optimizationcited by 138
  6. 2023Relightable 3D Gaussians: Realistic Point Cloud Relighting with BRDF Decomposition and Ray Tracingcited by 138
  7. 2023TextMesh: Generation of Realistic 3D Meshes From Text Promptscited by 135
  8. 2023Relightable Gaussian Codec Avatarscited by 126
  9. 2023StreetSurf: Extending Multi-view Implicit Surface Reconstruction to Street Viewscited by 120
  10. 2023F2-NeRF: Fast Neural Radiance Field Training with Free Camera Trajectoriescited by 118
  11. 2022MagicPony: Learning Articulated 3D Animals in the Wildcited by 111
  12. 2023Delicate Textured Mesh Recovery from NeRF via Adaptive Surface Refinementcited by 106
  13. 2022NeILF: Neural Incident Light Field for Physically-based Material Estimationcited by 105
  14. 2022Compressible-composable NeRF via Rank-residual Decompositioncited by 92
  15. 2022SAMURAI: Shape And Material from Unconstrained Real-world Arbitrary Image collectionscited by 89
  16. 2023Neural Fields meet Explicit Geometric Representation for Inverse Rendering of Urban Scenescited by 89
  17. 2023GIR: 3D Gaussian Inverse Rendering for Relightable Scene Factorizationcited by 86
  18. 2022TANGO: Text-driven Photorealistic and Robust 3D Stylization via Lighting Decompositioncited by 83
  19. 2022Learning-based Inverse Rendering of Complex Indoor Scenes with Differentiable Monte Carlo Raytracingcited by 78
  20. 2023FocalDreamer: Text-driven 3D Editing via Focal-fusion Assemblycited by 75
  21. 2023NeILF++: Inter-Reflectable Light Fields for Geometry and Material Estimationcited by 73
  22. 2024Spec-Gaussian: Anisotropic View-Dependent Appearance for 3D Gaussian Splattingcited by 73
  23. 2023BOP Challenge 2022 on Detection, Segmentation and Pose Estimation of Specific Rigid Objectscited by 70
  24. 2022DM-NeRF: 3D Scene Geometry Decomposition and Manipulation from 2D Imagescited by 69
  25. 2023Adaptive Shells for Efficient Neural Radiance Field Renderingcited by 69
  26. 2024IntrinsicAnything: Learning Diffusion Priors for Inverse Rendering Under Unknown Illuminationcited by 67
  27. 2024Meta 3D AssetGen: Text-to-Mesh Generation with High-Quality Geometry, Texture, and PBR Materialscited by 66
  28. 2024SF3D: Stable Fast 3D Mesh Reconstruction with UV-unwrapping and Illumination Disentanglementcited by 65
  29. 2022Human Performance Modeling and Rendering via Neural Animated Meshcited by 64
  30. 2023ENVIDR: Implicit Differentiable Renderer with Neural Environment Lightingcited by 63
  31. 20243D Gaussian Splatting with Deferred Reflectioncited by 61
  32. 2022Interactive Segmentation of Radiance Fieldscited by 60
  33. 2024BOP Challenge 2023 on Detection, Segmentation and Pose Estimation of Seen and Unseen Rigid Objectscited by 60
  34. 2024FlashTex: Fast Relightable Mesh Texturing with LightControlNetcited by 56
  35. 2023Urban Radiance Field Representation with Deformable Neural Mesh Primitivescited by 55
  36. 2025DiffusionRenderer: Neural Inverse and Forward Rendering with Video Diffusion Modelscited by 55
  37. 2023NeRFMeshing: Distilling Neural Radiance Fields into Geometrically-Accurate 3D Meshescited by 52
  38. 2023Stanford-ORB: A Real-World 3D Object Inverse Rendering Benchmarkcited by 51

50 most cited of 363 All 363 (JSON)