# Shape and Reflectance Reconstruction in Uncontrolled Environments by Differentiable Rendering

arXiv: https://arxiv.org/abs/2110.12975 · HTML: https://mlsanity.com/paper/2110.12975/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 58 references that are in the map.
- First posted on arXiv 25 Oct 2021.

## Builds on (most cited first, 25 of 25)

- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [Semantic Understanding of Scenes through the ADE20K Dataset](https://mlsanity.com/paper/1608.05442/) (2016)
- [Neural Sparse Voxel Fields](https://mlsanity.com/paper/2007.11571/) (2020)
- [Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations](https://mlsanity.com/paper/1906.01618/) (2019)
- [Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning](https://mlsanity.com/paper/1904.01786/) (2019)
- [DeepVoxels: Learning Persistent 3D Feature Embeddings](https://mlsanity.com/paper/1812.01024/) (2018)
- [SynSin: End-to-end View Synthesis from a Single Image](https://mlsanity.com/paper/1912.08804/) (2019)
- [Neural Point-Based Graphics](https://mlsanity.com/paper/1906.08240/) (2019)
- [Deferred Neural Rendering: Image Synthesis using Neural Textures](https://mlsanity.com/paper/1904.12356/) (2019)
- [DeepView: View Synthesis with Learned Gradient Descent](https://mlsanity.com/paper/1906.07316/) (2019)
- [Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling](https://mlsanity.com/paper/1804.04610/) (2018)
- [AutoInt: Automatic Integration for Fast Neural Volume Rendering](https://mlsanity.com/paper/2012.01714/) (2020)
- [Inverse Rendering for Complex Indoor Scenes: Shape, Spatially-Varying Lighting and SVBRDF from a Single Image](https://mlsanity.com/paper/1905.02722/) (2019)
- [Pushing the Boundaries of View Extrapolation with Multiplane Images](https://mlsanity.com/paper/1905.00413/) (2019)
- [DeepStereo: Learning to Predict New Views from the World's Imagery](https://mlsanity.com/paper/1506.06825/) (2015)
- [Neural Rerendering in the Wild](https://mlsanity.com/paper/1904.04290/) (2019)
- [Deep Parametric Indoor Lighting Estimation](https://mlsanity.com/paper/1910.08812/) (2019)
- [Unified Shape and SVBRDF Recovery using Differentiable Monte Carlo Rendering](https://mlsanity.com/paper/2103.15208/) (2021)
- [Neural Point Cloud Rendering via Multi-Plane Projection](https://mlsanity.com/paper/1912.04645/) (2019)
- [Full 3D Reconstruction of Transparent Objects](https://mlsanity.com/paper/1805.03482/) (2018)
- [LookinGood: Enhancing Performance Capture with Real-time Neural Re-Rendering](https://mlsanity.com/paper/1811.05029/) (2018)
- [Neural Human Video Rendering by Learning Dynamic Textures and Rendering-to-Video Translation](https://mlsanity.com/paper/2001.04947/) (2020)
- [Deep Lighting Environment Map Estimation from Spherical Panoramas](https://mlsanity.com/paper/2005.08000/) (2020)

## Built on by (most cited first, 1 of 1)

- [Efficient Multi-View Inverse Rendering Using a Hybrid Differentiable Rendering Method](https://mlsanity.com/paper/2308.10003/) (2023)

## Abstract (arXiv)

Simultaneous reconstruction of geometry and reflectance properties in uncontrolled environments remains a challenging problem. In this paper, we propose an efficient method to reconstruct the scene's 3D geometry and reflectance from multi-view photography using conventional hand-held cameras. Our method automatically builds a virtual scene in a differentiable rendering system that roughly matches the real world's scene parameters, optimized by minimizing photometric objectives alternatingly and stochastically. With the optimal scene parameters evaluated, photo-realistic novel views for various viewing angles and distances can then be generated by our approach. We present the results of captured scenes with complex geometry and various reflection types. Our method also shows superior performance compared to state-of-the-art alternatives in novel view synthesis visually and quantitatively.