# Tree-Structured Shading Decomposition

arXiv: https://arxiv.org/abs/2309.07122 · HTML: https://mlsanity.com/paper/2309.07122/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 61 references that are in the map.
- First posted on arXiv 13 Sep 2023.

## Builds on (most cited first, 30 of 31)

- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [NeRD: Neural Reflectance Decomposition from Image Collections](https://mlsanity.com/paper/2012.03918/) (2020)
- [Extracting Triangular 3D Models, Materials, and Lighting From Images](https://mlsanity.com/paper/2111.12503/) (2021)
- [PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relighting](https://mlsanity.com/paper/2104.00674/) (2021)
- [Inverse Rendering for Complex Indoor Scenes: Shape, Spatially-Varying Lighting and SVBRDF from a Single Image](https://mlsanity.com/paper/1905.02722/) (2019)
- [TensoIR: Tensorial Inverse Rendering](https://mlsanity.com/paper/2304.12461/) (2023)
- [Shape, Light, and Material Decomposition from Images using Monte Carlo Rendering and Denoising](https://mlsanity.com/paper/2206.03380/) (2022)
- [PixelSNAIL: An Improved Autoregressive Generative Model](https://mlsanity.com/paper/1712.09763/) (2017)
- [Single-Image SVBRDF Capture with a Rendering-Aware Deep Network](https://mlsanity.com/paper/1810.09718/) (2018)
- [IRON: Inverse Rendering by Optimizing Neural SDFs and Materials from Photometric Images](https://mlsanity.com/paper/2204.02232/) (2022)
- [CSGNet: Neural Shape Parser for Constructive Solid Geometry](https://mlsanity.com/paper/1712.08290/) (2017)
- [Transforming and Projecting Images into Class-conditional Generative Networks](https://mlsanity.com/paper/2005.01703/) (2020)
- [Learning Intrinsic Image Decomposition from Watching the World](https://mlsanity.com/paper/1804.00582/) (2018)
- [Unified Shape and SVBRDF Recovery using Differentiable Monte Carlo Rendering](https://mlsanity.com/paper/2103.15208/) (2021)
- [Learning to Infer and Execute 3D Shape Programs](https://mlsanity.com/paper/1901.02875/) (2019)
- [Unsupervised Learning for Intrinsic Image Decomposition from a Single Image](https://mlsanity.com/paper/1911.09930/) (2019)
- [Self-Supervised Intrinsic Image Decomposition](https://mlsanity.com/paper/1711.03678/) (2017)
- [A Survey on Intrinsic Images: Delving Deep Into Lambert and Beyond](https://mlsanity.com/paper/2112.03842/) (2021)
- [Generative Modelling of BRDF Textures from Flash Images](https://mlsanity.com/paper/2102.11861/) (2021)
- [TileGen: Tileable, Controllable Material Generation and Capture](https://mlsanity.com/paper/2206.05649/) (2022)
- [An Inverse Procedural Modeling Pipeline for SVBRDF Maps](https://mlsanity.com/paper/2109.06395/) (2021)
- [PLAD: Learning to Infer Shape Programs with Pseudo-Labels and Approximate Distributions](https://mlsanity.com/paper/2011.13045/) (2020)
- [RIM-Net: Recursive Implicit Fields for Unsupervised Learning of Hierarchical Shape Structures](https://mlsanity.com/paper/2201.12763/) (2022)
- [Deep Reflectance Maps](https://mlsanity.com/paper/1511.04384/) (2015)
- [Node Graph Optimization Using Differentiable Proxies](https://mlsanity.com/paper/2207.07684/) (2022)
- [Program-Guided Image Manipulators](https://mlsanity.com/paper/1909.02116/) (2019)
- [MatFormer: A Generative Model for Procedural Materials](https://mlsanity.com/paper/2207.01044/) (2022)
- [Controlling Material Appearance by Examples](https://mlsanity.com/paper/2206.14970/) (2022)
- [Perspective Plane Program Induction from a Single Image](https://mlsanity.com/paper/2006.14708/) (2020)
- [Inverse Procedural Modeling of Facade Layouts](https://mlsanity.com/paper/1308.0419/) (2013)

## Built on by (most cited first, 2 of 2)

- [ZeroComp: Zero-shot Object Compositing from Image Intrinsics via Diffusion](https://mlsanity.com/paper/2410.08168/) (2024)
- [Ouroboros: Single-step Diffusion Models for Cycle-consistent Forward and Inverse Rendering](https://mlsanity.com/paper/2508.14461/) (2025)

## Abstract (arXiv)

We study inferring a tree-structured representation from a single image for object shading. Prior work typically uses the parametric or measured representation to model shading, which is neither interpretable nor easily editable. We propose using the shade tree representation, which combines basic shading nodes and compositing methods to factorize object surface shading. The shade tree representation enables novice users who are unfamiliar with the physical shading process to edit object shading in an efficient and intuitive manner. A main challenge in inferring the shade tree is that the inference problem involves both the discrete tree structure and the continuous parameters of the tree nodes. We propose a hybrid approach to address this issue. We introduce an auto-regressive inference model to generate a rough estimation of the tree structure and node parameters, and then we fine-tune the inferred shade tree through an optimization algorithm. We show experiments on synthetic images, captured reflectance, real images, and non-realistic vector drawings, allowing downstream applications such as material editing, vectorized shading, and relighting. Project website: https://chen-geng.com/inv-shade-trees