# Intrinsic3D: High-Quality 3D Reconstruction by Joint Appearance and Geometry Optimization with Spatially-Varying Lighting

arXiv: https://arxiv.org/abs/1708.01670 · HTML: https://mlsanity.com/paper/1708.01670/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Aug 2017.

## Built on by (most cited first, 30 of 31)

- [PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relighting](https://mlsanity.com/paper/2104.00674/) (2021)
- [AKB-48: A Real-World Articulated Object Knowledge Base](https://mlsanity.com/paper/2202.08432/) (2022)
- [Unified Shape and SVBRDF Recovery using Differentiable Monte Carlo Rendering](https://mlsanity.com/paper/2103.15208/) (2021)
- [StyleLight: HDR Panorama Generation for Lighting Estimation and Editing](https://mlsanity.com/paper/2207.14811/) (2022)
- [Deep Implicit Surface Point Prediction Networks](https://mlsanity.com/paper/2106.05779/) (2021)
- [DiffComplete: Diffusion-based Generative 3D Shape Completion](https://mlsanity.com/paper/2306.16329/) (2023)
- [NeMF: Inverse Volume Rendering with Neural Microflake Field](https://mlsanity.com/paper/2304.00782/) (2023)
- [Neural Material Adaptor for Visual Grounding of Intrinsic Dynamics](https://mlsanity.com/paper/2410.08257/) (2024)
- [GMLight: Lighting Estimation via Geometric Distribution Approximation](https://mlsanity.com/paper/2102.10244/) (2021)
- [Neural Radiance Fields Approach to Deep Multi-View Photometric Stereo](https://mlsanity.com/paper/2110.05594/) (2021)
- [Factorized Inverse Path Tracing for Efficient and Accurate Material-Lighting Estimation](https://mlsanity.com/paper/2304.05669/) (2023)
- [WildLight: In-the-wild Inverse Rendering with a Flashlight](https://mlsanity.com/paper/2303.14190/) (2023)
- [Sparse Needlets for Lighting Estimation with Spherical Transport Loss](https://mlsanity.com/paper/2106.13090/) (2021)
- [MAIR: Multi-view Attention Inverse Rendering with 3D Spatially-Varying Lighting Estimation](https://mlsanity.com/paper/2303.12368/) (2023)
- [ASH: A Modern Framework for Parallel Spatial Hashing in 3D Perception](https://mlsanity.com/paper/2110.00511/) (2021)
- [Deep Graph Learning for Spatially-Varying Indoor Lighting Prediction](https://mlsanity.com/paper/2202.06300/) (2022)
- [Free-form 3D Scene Inpainting with Dual-stream GAN](https://mlsanity.com/paper/2212.08464/) (2022)
- [Adaptive Joint Optimization for 3D Reconstruction with Differentiable Rendering](https://mlsanity.com/paper/2208.07003/) (2022)
- [High-Quality RGB-D Reconstruction via Multi-View Uncalibrated Photometric Stereo and Gradient-SDF](https://mlsanity.com/paper/2210.12202/) (2022)
- [TwinTex: Geometry-aware Texture Generation for Abstracted 3D Architectural Models](https://mlsanity.com/paper/2309.11258/) (2023)
- [One Ring to Rule Them All: a simple solution to multi-view 3D-Reconstruction of shapes with unknown BRDF via a small Recurrent ResNet](https://mlsanity.com/paper/2104.05014/) (2021)
- [Dynamic Voxel Grid Optimization for High-Fidelity RGB-D Supervised Surface Reconstruction](https://mlsanity.com/paper/2304.06178/) (2023)
- [Towards Scalable Multi-View Reconstruction of Geometry and Materials](https://mlsanity.com/paper/2306.03747/) (2023)
- [MAIR++: Improving Multi-view Attention Inverse Rendering with Implicit Lighting Representation](https://mlsanity.com/paper/2408.06707/) (2024)
- [Spatially and color consistent environment lighting estimation using deep neural networks for mixed reality](https://mlsanity.com/paper/2108.07903/) (2021)
- [LitAR: Visually Coherent Lighting for Mobile Augmented Reality](https://mlsanity.com/paper/2301.06184/) (2023)
- [Spin-UP: Spin Light for Natural Light Uncalibrated Photometric Stereo](https://mlsanity.com/paper/2404.01612/) (2024)
- [Aerial Path Planning for Urban Geometry and Texture Co-Capture](https://mlsanity.com/paper/2509.22227/) (2025)
- [VQT-Light:Lightweight HDR Illumination Map Prediction with Richer Texture.pdf](https://mlsanity.com/paper/2509.12556/) (2025)
- [DiffTex: Differentiable Texturing for Architectural Proxy Models](https://mlsanity.com/paper/2509.23336/) (2025)

## Abstract (arXiv)

We introduce a novel method to obtain high-quality 3D reconstructions from consumer RGB-D sensors. Our core idea is to simultaneously optimize for geometry encoded in a signed distance field (SDF), textures from automatically-selected keyframes, and their camera poses along with material and scene lighting. To this end, we propose a joint surface reconstruction approach that is based on Shape-from-Shading (SfS) techniques and utilizes the estimation of spatially-varying spherical harmonics (SVSH) from subvolumes of the reconstructed scene. Through extensive examples and evaluations, we demonstrate that our method dramatically increases the level of detail in the reconstructed scene geometry and contributes highly to consistent surface texture recovery.