mlsanityData to 14 Oct 2026

PSDR-Room: Single Photo to Scene using Differentiable Rendering

arXiv 2307.03244 ↗First posted 6 Jul 2023cs.CV cs.GRcited by 23

Not on the map. Why?

Abstract (arXiv)

A 3D digital scene contains many components: lights, materials and geometries, interacting to reach the desired appearance. Staging such a scene is time-consuming and requires both artistic and technical skills. In this work, we propose PSDR-Room, a system allowing to optimize lighting as well as the pose and materials of individual objects to match a target image of a room scene, with minimal user input. To this end, we leverage a recent path-space differentiable rendering approach that provides unbiased gradients of the rendering with respect to geometry, lighting, and procedural materials, allowing us to optimize all of these components using gradient descent to visually match the input photo appearance. We use recent single-image scene understanding methods to initialize the optimization and search for appropriate 3D models and materials. We evaluate our method on real photographs of indoor scenes and demonstrate the editability of the resulting scene components.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Builds on 24 of its 50 references that are in the map. First posted on arXiv 6 Jul 2023.

Builds on

24 of 50 references.

References · 24

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  4. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  5. 2023Segment Anythingcited by 8,234
  6. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,351
  7. 2021Vision Transformers for Dense Predictioncited by 1,803
  8. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,495
  9. 2019Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoningcited by 420
  10. 20203D-FUTURE: 3D Furniture shape with TextUREcited by 311
  11. 2019Inverse Rendering for Complex Indoor Scenes: Shape, Spatially-Varying Lighting and SVBRDF from a Single Imagecited by 203
  12. 2020Total3DUnderstanding: Joint Layout, Object Pose and Mesh Reconstruction for Indoor Scenes from a Single Imagecited by 172
12 more
  1. 2016IM2CADcited by 78
  2. 2022Perspective Fields for Single Image Camera Calibrationcited by 75
  3. 2021Unified Shape and SVBRDF Recovery using Differentiable Monte Carlo Renderingcited by 71
  4. 2019Inverse Path Tracing for Joint Material and Lighting Estimationcited by 62
  5. 2022IRISformer: Dense Vision Transformers for Single-Image Inverse Rendering in Indoor Scenescited by 62
  6. 2018Cooperative Holistic Scene Understanding: Unifying 3D Object, Layout, and Camera Pose Estimationcited by 51
  7. 2021An Inverse Procedural Modeling Pipeline for SVBRDF Mapscited by 24
  8. 2022PhotoScene: Photorealistic Material and Lighting Transfer for Indoor Scenescited by 22
  9. 2022Node Graph Optimization Using Differentiable Proxiescited by 13
  10. 2022MatFormer: A Generative Model for Procedural Materialscited by 12
  11. 2022Controlling Material Appearance by Examplescited by 10
  12. 2022Photo-to-Shape Material Transfer for Diverse Structurescited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 7; 2025: 9; 2026: 6.

23 papers.

Sort:

By area:

  1. 2024Gen3DSR: Generalizable 3D Scene Reconstruction via Divide and Conquer from a Single Viewcited by 61
  2. 2024TextureDreamer: Image-guided Texture Synthesis through Geometry-aware Diffusioncited by 52
  3. 2024MatSynth: A Modern PBR Materials Datasetcited by 42
  4. 2024MaPa: Text-driven Photorealistic Material Painting for 3D Shapescited by 27
  5. 2025DepR: Depth Guided Single-view Scene Reconstruction with Instance-level Diffusioncited by 23
  6. 2025LiteReality: Graphics-Ready 3D Scene Reconstruction from RGB-D Scanscited by 21
  7. 2024MatAtlas: Text-driven Consistent Geometry Texturing and Material Assignmentcited by 14
  8. 20263D-Fixer: Coarse-to-Fine In-place Completion for 3D Scenes from a Single Imagecited by 11
  9. 2024MaterialPicker: Multi-Modal DiT-Based Material Generationcited by 10
  10. 2025Materialist: Physically Based Editing Using Single-Image Inverse Renderingcited by 10
  11. 2024UrbanCAD: Towards Highly Controllable and Photorealistic 3D Vehicles for Urban Scene Simulationcited by 7
  12. 2025Digital Twin Generation from Visual Data: A Surveycited by 7
11 more
  1. 2025MatSwap: Light-aware material transfers in imagescited by 6
  2. 2023Zero Grads: Learning Local Surrogate Losses for Non-Differentiable Graphicscited by 5
  3. 2025HLG: Comprehensive 3D Room Construction via Hierarchical Layout Generationcited by 4
  4. 2026AdaSFormer: Adaptive Serialized Transformers for Monocular Semantic Scene Completion from Indoor Environmentscited by 3
  5. 2025MatCLIP: Light- and Shape-Insensitive Assignment of PBR Material Modelscited by 2
  6. 2026Seeing Through Clutter: Structured 3D Scene Reconstruction via Iterative Object Removalcited by 2
  7. 2025RealMat: Realistic Materials with Diffusion and Reinforcement Learningcited by 1
  8. 2026EAG-PT: Emission-Aware Gaussians and Path Tracing for Diffuse Indoor Scene Reconstruction and Editingcited by 1
  9. 2025An evaluation of SVBRDF Prediction from Generative Image Models for Appearance Modeling of 3D Scenescited by 0
  10. 2026DecoRec: Decomposed 3D Scene Reconstruction from Single-View Images via Object-Level Diffusioncited by 0
  11. 2026LiteReality-Agent: An Agentic System for Interactable 3D Indoor Scene Reconstructioncited by 0