# Multi-view Gradient Consistency for SVBRDF Estimation of Complex Scenes under Natural Illumination

arXiv: https://arxiv.org/abs/2202.13017 · HTML: https://mlsanity.com/paper/2202.13017/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 36 references that are in the map.
- First posted on arXiv 25 Feb 2022.

## Builds on (most cited first, 12 of 12)

- [MVSNet: Depth Inference for Unstructured Multi-view Stereo](https://mlsanity.com/paper/1804.02505/) (2018)
- [Accelerating 3D Deep Learning with PyTorch3D](https://mlsanity.com/paper/2007.08501/) (2020)
- [BlendedMVS: A Large-scale Dataset for Generalized Multi-view Stereo Networks](https://mlsanity.com/paper/1911.10127/) (2019)
- [Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer](https://mlsanity.com/paper/1908.01210/) (2019)
- [Visual Object Networks: Image Generation with Disentangled 3D Representation](https://mlsanity.com/paper/1812.02725/) (2018)
- [Neural Rerendering in the Wild](https://mlsanity.com/paper/1904.04290/) (2019)
- [Image GANs meet Differentiable Rendering for Inverse Graphics and Interpretable 3D Neural Rendering](https://mlsanity.com/paper/2010.09125/) (2020)
- [Unified Shape and SVBRDF Recovery using Differentiable Monte Carlo Rendering](https://mlsanity.com/paper/2103.15208/) (2021)
- [Materials for Masses: SVBRDF Acquisition with a Single Mobile Phone Image](https://mlsanity.com/paper/1804.05790/) (2018)
- [Modeling Surface Appearance from a Single Photograph using Self-augmented Convolutional Neural Networks](https://mlsanity.com/paper/1809.00886/) (2018)
- [A Novel Recurrent Encoder-Decoder Structure for Large-Scale Multi-view Stereo Reconstruction from An Open Aerial Dataset](https://mlsanity.com/paper/2003.00637/) (2020)
- [Shape from Shading through Shape Evolution](https://mlsanity.com/paper/1712.02961/) (2017)

## Built on by (most cited first, 3 of 3)

- [IntrinsicNeRF: Learning Intrinsic Neural Radiance Fields for Editable Novel View Synthesis](https://mlsanity.com/paper/2210.00647/) (2022)
- [Material Palette: Extraction of Materials from a Single Image](https://mlsanity.com/paper/2311.17060/) (2023)
- [VQ-NeRF: Neural Reflectance Decomposition and Editing with Vector Quantization](https://mlsanity.com/paper/2310.11864/) (2023)

## Abstract (arXiv)

This paper presents a process for estimating the spatially varying surface reflectance of complex scenes observed under natural illumination. In contrast to previous methods, our process is not limited to scenes viewed under controlled lighting conditions but can handle complex indoor and outdoor scenes viewed under arbitrary illumination conditions. An end-to-end process uses a model of the scene's geometry and several images capturing the scene's surfaces from arbitrary viewpoints and under various natural illumination conditions. We develop a differentiable path tracer that leverages least-square conformal mapping for handling multiple disjoint objects appearing in the scene. We follow a two-step optimization process and introduce a multi-view gradient consistency loss which results in up to 30-50% improvement in the image reconstruction loss and can further achieve better disentanglement of the diffuse and specular BRDFs compared to other state-of-the-art. We demonstrate the process in real-world indoor and outdoor scenes from images in the wild and show that we can produce realistic renders consistent with actual images using the estimated reflectance properties. Experiments show that our technique produces realistic results for arbitrary outdoor scenes with complex geometry. The source code is publicly available at: https://gitlab.com/alen.joy/multi-view-gradient-consistency-for-svbrdf-estimation-of-complex-scenes-under-natural-illumination