# Modeling Surface Appearance from a Single Photograph using Self-augmented Convolutional Neural Networks

arXiv: https://arxiv.org/abs/1809.00886 · HTML: https://mlsanity.com/paper/1809.00886/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Sep 2018.

## Built on by (most cited first, 30 of 51)

- [ABO: Dataset and Benchmarks for Real-World 3D Object Understanding](https://mlsanity.com/paper/2110.06199/) (2021)
- [Neural-PIL: Neural Pre-Integrated Lighting for Reflectance Decomposition](https://mlsanity.com/paper/2110.14373/) (2021)
- [SAMURAI: Shape And Material from Unconstrained Real-world Arbitrary Image collections](https://mlsanity.com/paper/2205.15768/) (2022)
- [Physically-Based Editing of Indoor Scene Lighting from a Single Image](https://mlsanity.com/paper/2205.09343/) (2022)
- [Deep Polarization Imaging for 3D shape and SVBRDF Acquisition](https://mlsanity.com/paper/2105.02875/) (2021)
- [Free-viewpoint Indoor Neural Relighting from Multi-view Stereo](https://mlsanity.com/paper/2106.13299/) (2021)
- [ControlMat: A Controlled Generative Approach to Material Capture](https://mlsanity.com/paper/2309.01700/) (2023)
- [AvatarMe++: Facial Shape and BRDF Inference with Photorealistic Rendering-Aware GANs](https://mlsanity.com/paper/2112.05957/) (2021)
- [MatSynth: A Modern PBR Materials Dataset](https://mlsanity.com/paper/2401.06056/) (2024)
- [MatFuse: Controllable Material Generation with Diffusion Models](https://mlsanity.com/paper/2308.11408/) (2023)
- [A Survey on Intrinsic Images: Delving Deep Into Lambert and Beyond](https://mlsanity.com/paper/2112.03842/) (2021)
- [Generative Modelling of BRDF Textures from Flash Images](https://mlsanity.com/paper/2102.11861/) (2021)
- [MatFusion: A Generative Diffusion Model for SVBRDF Capture](https://mlsanity.com/paper/2406.06539/) (2024)
- [Material Palette: Extraction of Materials from a Single Image](https://mlsanity.com/paper/2311.17060/) (2023)
- [TileGen: Tileable, Controllable Material Generation and Capture](https://mlsanity.com/paper/2206.05649/) (2022)
- [SurfaceNet: Adversarial SVBRDF Estimation from a Single Image](https://mlsanity.com/paper/2107.11298/) (2021)
- [SP-GAN: Sphere-Guided 3D Shape Generation and Manipulation](https://mlsanity.com/paper/2108.04476/) (2021)
- [IRIS: Inverse Rendering of Indoor Scenes from Low Dynamic Range Images](https://mlsanity.com/paper/2401.12977/) (2024)
- [Outdoor inverse rendering from a single image using multiview self-supervision](https://mlsanity.com/paper/2102.06591/) (2021)
- [PhotoMat: A Material Generator Learned from Single Flash Photos](https://mlsanity.com/paper/2305.12296/) (2023)
- [Plan2Scene: Converting Floorplans to 3D Scenes](https://mlsanity.com/paper/2106.05375/) (2021)
- [LIRM: Large Inverse Rendering Model for Progressive Reconstruction of Shape, Materials and View-dependent Radiance Fields](https://mlsanity.com/paper/2504.20026/) (2025)
- [Active Exploration for Neural Global Illumination of Variable Scenes](https://mlsanity.com/paper/2203.08272/) (2022)
- [UMat: Uncertainty-Aware Single Image High Resolution Material Capture](https://mlsanity.com/paper/2305.16312/) (2023)
- [Inverse Rendering of Translucent Objects using Physical and Neural Renderers](https://mlsanity.com/paper/2305.08336/) (2023)
- [VLMaterial: Procedural Material Generation with Large Vision-Language Models](https://mlsanity.com/paper/2501.18623/) (2025)
- [VQ-NeRF: Neural Reflectance Decomposition and Editing with Vector Quantization](https://mlsanity.com/paper/2310.11864/) (2023)
- [Neural Photometry-guided Visual Attribute Transfer](https://mlsanity.com/paper/2112.02520/) (2021)
- [MaterialSeg3D: Segmenting Dense Materials from 2D Priors for 3D Assets](https://mlsanity.com/paper/2404.13923/) (2024)
- [Deep Polarization Cues for Single-shot Shape and Subsurface Scattering Estimation](https://mlsanity.com/paper/2407.08149/) (2024)

## Abstract (arXiv)

We present a convolutional neural network (CNN) based solution for modeling physically plausible spatially varying surface reflectance functions (SVBRDF) from a single photograph of a planar material sample under unknown natural illumination. Gathering a sufficiently large set of labeled training pairs consisting of photographs of SVBRDF samples and corresponding reflectance parameters, is a difficult and arduous process. To reduce the amount of required labeled training data, we propose to leverage the appearance information embedded in unlabeled images of spatially varying materials to self-augment the training process. Starting from an initial approximative network obtained from a small set of labeled training pairs, we estimate provisional model parameters for each unlabeled training exemplar. Given this provisional reflectance estimate, we then synthesize a novel temporary labeled training pair by rendering the exact corresponding image under a new lighting condition. After refining the network using these additional training samples, we re-estimate the provisional model parameters for the unlabeled data and repeat the self-augmentation process until convergence. We demonstrate the efficacy of the proposed network structure on spatially varying wood, metals, and plastics, as well as thoroughly validate the effectiveness of the self-augmentation training process.