# Material Editing Using a Physically Based Rendering Network

arXiv: https://arxiv.org/abs/1708.00106 · HTML: https://mlsanity.com/paper/1708.00106/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Aug 2017.

## Built on by (most cited first, 17 of 17)

- [Alchemist: Parametric Control of Material Properties with Diffusion Models](https://mlsanity.com/paper/2312.02970/) (2023)
- [Shape and Material Capture at Home](https://mlsanity.com/paper/2104.06397/) (2021)
- [Generative Modelling of BRDF Textures from Flash Images](https://mlsanity.com/paper/2102.11861/) (2021)
- [De-rendering 3D Objects in the Wild](https://mlsanity.com/paper/2201.02279/) (2022)
- [De-rendering the World's Revolutionary Artefacts](https://mlsanity.com/paper/2104.03954/) (2021)
- [VoGE: A Differentiable Volume Renderer using Gaussian Ellipsoids for Analysis-by-Synthesis](https://mlsanity.com/paper/2205.15401/) (2022)
- [PhotoScene: Photorealistic Material and Lighting Transfer for Indoor Scenes](https://mlsanity.com/paper/2207.00757/) (2022)
- [Outdoor inverse rendering from a single image using multiview self-supervision](https://mlsanity.com/paper/2102.06591/) (2021)
- [SSPU-Net: Self-Supervised Point Cloud Upsampling via Differentiable Rendering](https://mlsanity.com/paper/2108.00454/) (2021)
- [GenDR: A Generalized Differentiable Renderer](https://mlsanity.com/paper/2204.13845/) (2022)
- [Learning to Learn and Sample BRDFs](https://mlsanity.com/paper/2210.03510/) (2022)
- [Inverse Rendering of Translucent Objects using Physical and Neural Renderers](https://mlsanity.com/paper/2305.08336/) (2023)
- [SeeClear: Reliable Transparent Object Depth Estimation via Generative Opacification](https://mlsanity.com/paper/2603.19547/) (2026)
- [Predicting Surface Reflectance Properties of Outdoor Scenes Under Unknown Natural Illumination](https://mlsanity.com/paper/2105.06820/) (2021)
- [StylePart: Image-based Shape Part Manipulation](https://mlsanity.com/paper/2111.10520/) (2021)
- [RetouchUAA: Unconstrained Adversarial Attack via Image Retouching](https://mlsanity.com/paper/2311.16478/) (2023)
- [Few-shot point cloud reconstruction and denoising via learned Guassian splats renderings and fine-tuned diffusion features](https://mlsanity.com/paper/2404.01112/) (2024)

## Abstract (arXiv)

The ability to edit materials of objects in images is desirable by many content creators. However, this is an extremely challenging task as it requires to disentangle intrinsic physical properties of an image. We propose an end-to-end network architecture that replicates the forward image formation process to accomplish this task. Specifically, given a single image, the network first predicts intrinsic properties, i.e. shape, illumination, and material, which are then provided to a rendering layer. This layer performs in-network image synthesis, thereby enabling the network to understand the physics behind the image formation process. The proposed rendering layer is fully differentiable, supports both diffuse and specular materials, and thus can be applicable in a variety of problem settings. We demonstrate a rich set of visually plausible material editing examples and provide an extensive comparative study.