mlsanityData to 7 Oct 2026

PhotoScene: Photorealistic Material and Lighting Transfer for Indoor Scenes

arXiv 2207.00757 ↗First posted 2 Jul 2022cs.CVcited by 22

Not on the map. Why?

Abstract (arXiv)

Most indoor 3D scene reconstruction methods focus on recovering 3D geometry and scene layout. In this work, we go beyond this to propose PhotoScene, a framework that takes input image(s) of a scene along with approximately aligned CAD geometry (either reconstructed automatically or manually specified) and builds a photorealistic digital twin with high-quality materials and similar lighting. We model scene materials using procedural material graphs; such graphs represent photorealistic and resolution-independent materials. We optimize the parameters of these graphs and their texture scale and rotation, as well as the scene lighting to best match the input image via a differentiable rendering layer. We evaluate our technique on objects and layout reconstructions from ScanNet, SUN RGB-D and stock photographs, and demonstrate that our method reconstructs high-quality, fully relightable 3D scenes that can be re-rendered under arbitrary viewpoints, zooms and lighting.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 33 of its 55 references that are in the map. First posted on arXiv 2 Jul 2022.

Builds on

33 of 55 references.

References · 33

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  4. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,148
  5. 2019StegaStamp: Invisible Hyperlinks in Physical Photographscited by 256
  6. 2019Recurrent MVSNet for High-resolution Multi-view Stereo Depth Inferencecited by 252
  7. 2019Inverse Rendering for Complex Indoor Scenes: Shape, Spatially-Varying Lighting and SVBRDF from a Single Imagecited by 202
  8. 2020Total3DUnderstanding: Joint Layout, Object Pose and Mesh Reconstruction for Indoor Scenes from a Single Imagecited by 170
  9. 2017MoFA: Model-based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstructioncited by 163
  10. 2018Single-Image SVBRDF Capture with a Rendering-Aware Deep Networkcited by 116
  11. 2017SfSNet: Learning Shape, Reflectance and Illuminance of Faces in the Wildcited by 110
  12. 2019HorizonNet: Learning Room Layout with 1D Representation and Pano Stretch Data Augmentationcited by 107
21 more
  1. 2019Neural Inverse Rendering of an Indoor Scene from a Single Imagecited by 101
  2. 2016IM2CADcited by 78
  3. 2014Material Recognition in the Wild with the Materials in Context Databasecited by 77
  4. 2020OpenRooms: An End-to-End Open Framework for Photorealistic Indoor Scene Datasetscited by 73
  5. 2019Rendering Synthetic Objects into Legacy Photographscited by 72
  6. 2017Neural Face Editing with Intrinsic Image Disentanglingcited by 70
  7. 2018Holistic 3D Scene Parsing and Reconstruction from a Single RGB Imagecited by 69
  8. 2018Materials for Masses: SVBRDF Acquisition with a Single Mobile Phone Imagecited by 66
  9. 2019Inverse Path Tracing for Joint Material and Lighting Estimationcited by 62
  10. 2019Flexible SVBRDF Capture with a Multi-Image Deep Networkcited by 50
  11. 2020SceneCAD: Predicting Object Alignments and Layouts in RGB-D Scanscited by 44
  12. 2019Automatic Scene Inference for 3D Object Compositingcited by 36
  13. 2021Generative Modelling of BRDF Textures from Flash Imagescited by 35
  14. 2018LIME: Live Intrinsic Material Estimationcited by 33
  15. 2020MaterialGAN: Reflectance Capture using a Generative SVBRDF Modelcited by 33
  16. 2021An Inverse Procedural Modeling Pipeline for SVBRDF Mapscited by 24
  17. 2018PhotoShape: Photorealistic Materials for Large-Scale Shape Collectionscited by 21
  18. 2017Material Editing Using a Physically Based Rendering Networkcited by 17
  19. 2021Plan2Scene: Converting Floorplans to 3D Scenescited by 17
  20. 2019A Bayesian Inference Framework for Procedural Material Parameter Estimationcited by 11
  21. 2018Layouts from Panoramic Images with Geometry and Deep Learningcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 8; 2024: 8; 2025: 4; 2026: 1.

22 papers.

Sort:

By area:

  1. 2024Gen3DSR: Generalizable 3D Scene Reconstruction via Divide and Conquer from a Single Viewcited by 59
  2. 2023Exploiting Diffusion Prior for Generalizable Dense Predictioncited by 37
  3. 2023Material Palette: Extraction of Materials from a Single Imagecited by 28
  4. 2024MaPa: Text-driven Photorealistic Material Painting for 3D Shapescited by 27
  5. 2023PSDR-Room: Single Photo to Scene using Differentiable Renderingcited by 23
  6. 2025LiteReality: Graphics-Ready 3D Scene Reconstruction from RGB-D Scanscited by 21
  7. 2023Text2Scene: Text-driven Indoor Scene Stylization with Part-aware Detailscited by 18
  8. 2024Objects With Lighting: A Real-World Dataset for Evaluating Reconstruction and Rendering for Object Relightingcited by 18
  9. 2023NeRFVS: Neural Radiance Fields for Free View Synthesis via Geometry Scaffoldscited by 16
  10. 2023MAIR: Multi-view Attention Inverse Rendering with 3D Spatially-Varying Lighting Estimationcited by 12
  11. 2025DNF-Intrinsic: Deterministic Noise-Free Diffusion for Indoor Inverse Renderingcited by 11
  12. 2024MaterialPicker: Multi-Modal DiT-Based Material Generationcited by 10
10 more
  1. 2024UrbanCAD: Towards Highly Controllable and Photorealistic 3D Vehicles for Urban Scene Simulationcited by 7
  2. 2024MaterialSeg3D: Segmenting Dense Materials from 2D Priors for 3D Assetscited by 6
  3. 2023Virtual Home Staging: Inverse Rendering and Editing an Indoor Panorama under Natural Illuminationcited by 5
  4. 2024MAIR++: Improving Multi-view Attention Inverse Rendering with Implicit Lighting Representationcited by 3
  5. 2023Seg2Reg: Differentiable 2D Segmentation to 1D Regression Rendering for 360 Room Layout Reconstructioncited by 2
  6. 2025MatCLIP: Light- and Shape-Insensitive Assignment of PBR Material Modelscited by 2
  7. 2024FabricDiffusion: High-Fidelity Texture Transfer for 3D Garments Generation from In-The-Wild Clothing Imagescited by 1
  8. 2022Deep scene-scale material estimation from multi-view indoor capturescited by 0
  9. 2025Indoor Heat Estimation from a Single Visible-Light Panoramacited by 0
  10. 2026LiteReality-Agent: An Agentic System for Interactable 3D Indoor Scene Reconstructioncited by 0