# Colorful Diffuse Intrinsic Image Decomposition in the Wild

arXiv: https://arxiv.org/abs/2409.13690 · HTML: https://mlsanity.com/paper/2409.13690/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 53 references that are in the map.
- First posted on arXiv 20 Sep 2024.

## Builds on (most cited first, 30 of 31)

- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [MegaDepth: Learning Single-View Depth Prediction from Internet Photos](https://mlsanity.com/paper/1804.00607/) (2018)
- [Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding](https://mlsanity.com/paper/2011.02523/) (2020)
- [Structured3D: A Large Photo-realistic Dataset for Structured 3D Modeling](https://mlsanity.com/paper/1908.00222/) (2019)
- [Inverse Rendering for Complex Indoor Scenes: Shape, Spatially-Varying Lighting and SVBRDF from a Single Image](https://mlsanity.com/paper/1905.02722/) (2019)
- [MatrixCity: A Large-scale City Dataset for City-scale Neural Rendering and Beyond](https://mlsanity.com/paper/2309.16553/) (2023)
- [RGBleftrightarrowX: Image decomposition and synthesis using material- and lighting-aware diffusion models](https://mlsanity.com/paper/2405.00666/) (2024)
- [Neural Inverse Rendering of an Indoor Scene from a Single Image](https://mlsanity.com/paper/1901.02453/) (2019)
- [CGIntrinsics: Better Intrinsic Image Decomposition through Physically-Based Rendering](https://mlsanity.com/paper/1808.08601/) (2018)
- [Learning Indoor Inverse Rendering with 3D Spatially-Varying Lighting](https://mlsanity.com/paper/2109.06061/) (2021)
- [Learning-based Inverse Rendering of Complex Indoor Scenes with Differentiable Monte Carlo Raytracing](https://mlsanity.com/paper/2211.03017/) (2022)
- [Learning Intrinsic Image Decomposition from Watching the World](https://mlsanity.com/paper/1804.00582/) (2018)
- [Learning to Relight Portrait Images via a Virtual Light Stage and Synthetic-to-Real Adaptation](https://mlsanity.com/paper/2209.10510/) (2022)
- [IntrinsicAnything: Learning Diffusion Priors for Inverse Rendering Under Unknown Illumination](https://mlsanity.com/paper/2404.11593/) (2024)
- [IRISformer: Dense Vision Transformers for Single-Image Inverse Rendering in Indoor Scenes](https://mlsanity.com/paper/2206.08423/) (2022)
- [Unsupervised Learning for Intrinsic Image Decomposition from a Single Image](https://mlsanity.com/paper/1911.09930/) (2019)
- [Intrinsic Image Decomposition via Ordinal Shading](https://mlsanity.com/paper/2311.12792/) (2023)
- [Self-Supervised Intrinsic Image Decomposition](https://mlsanity.com/paper/1711.03678/) (2017)
- [Learning Non-Lambertian Object Intrinsics across ShapeNet Categories](https://mlsanity.com/paper/1612.08510/) (2016)
- [A Survey on Intrinsic Images: Delving Deep Into Lambert and Beyond](https://mlsanity.com/paper/2112.03842/) (2021)
- [Convolutional Color Constancy](https://mlsanity.com/paper/1507.00410/) (2015)
- [Automatic Scene Inference for 3D Object Compositing](https://mlsanity.com/paper/1912.12297/) (2019)
- [LIME: Live Intrinsic Material Estimation](https://mlsanity.com/paper/1801.01075/) (2018)
- [EDEN: Multimodal Synthetic Dataset of Enclosed GarDEN Scenes](https://mlsanity.com/paper/2011.04389/) (2020)
- [Joint Learning of Intrinsic Images and Semantic Segmentation](https://mlsanity.com/paper/1807.11857/) (2018)
- [PIE-Net: Photometric Invariant Edge Guided Network for Intrinsic Image Decomposition](https://mlsanity.com/paper/2203.16670/) (2022)
- [Unsupervised Deep Single-Image Intrinsic Decomposition using Illumination-Varying Image Sequences](https://mlsanity.com/paper/1803.00805/) (2018)
- [CNN based Learning using Reflection and Retinex Models for Intrinsic Image Decomposition](https://mlsanity.com/paper/1712.01056/) (2017)
- [Real-Time Global Illumination Decomposition of Videos](https://mlsanity.com/paper/1908.01961/) (2019)
- [Measured Albedo in the Wild: Filling the Gap in Intrinsics Evaluation](https://mlsanity.com/paper/2306.15662/) (2023)

## Built on by (most cited first, 30 of 44)

- [Marigold: Affordable Adaptation of Diffusion-Based Image Generators for Image Analysis](https://mlsanity.com/paper/2505.09358/) (2025)
- [LumiNet: Latent Intrinsics Meets Diffusion Models for Indoor Scene Relighting](https://mlsanity.com/paper/2412.00177/) (2024)
- [ScribbleLight: Single Image Indoor Relighting with Scribbles](https://mlsanity.com/paper/2411.17696/) (2024)
- [IntrinsiX: High-Quality PBR Generation using Image Priors](https://mlsanity.com/paper/2504.01008/) (2025)
- [Ouroboros: Single-step Diffusion Models for Cycle-consistent Forward and Inverse Rendering](https://mlsanity.com/paper/2508.14461/) (2025)
- [Physically Controllable Relighting of Photographs](https://mlsanity.com/paper/2508.05626/) (2025)
- [DNF-Intrinsic: Deterministic Noise-Free Diffusion for Indoor Inverse Rendering](https://mlsanity.com/paper/2507.03924/) (2025)
- [Generative World Renderer](https://mlsanity.com/paper/2604.02329/) (2026)
- [PRISM: A Unified Framework for Photorealistic Reconstruction and Intrinsic Scene Modeling](https://mlsanity.com/paper/2504.14219/) (2025)
- [PractiLight: Practical Light Control Using Foundational Diffusion Models](https://mlsanity.com/paper/2509.01837/) (2025)
- [FlowPortal: Residual-Corrected Flow for Training-Free Video Relighting and Background Replacement](https://mlsanity.com/paper/2511.18346/) (2025)
- [MVInverse: Feed-forward Multi-view Inverse Rendering in Seconds](https://mlsanity.com/paper/2512.21003/) (2025)
- [Jodi: Unification of Visual Generation and Understanding via Joint Modeling](https://mlsanity.com/paper/2505.19084/) (2025)
- [Chord: Chain of Rendering Decomposition for PBR Material Estimation from Generated Texture Images](https://mlsanity.com/paper/2509.09952/) (2025)
- [LightCity: An Urban Dataset for Outdoor Inverse Rendering and Reconstruction under Multi-illumination Conditions](https://mlsanity.com/paper/2602.01118/) (2026)
- [ePBR: Extended PBR Materials in Image Synthesis](https://mlsanity.com/paper/2504.17062/) (2025)
- [UniVidX: A Unified Multimodal Framework for Versatile Video Generation via Diffusion Priors](https://mlsanity.com/paper/2605.00658/) (2026)
- [StableMTL: Repurposing Latent Diffusion Models for Multi-Task Learning from Partially Annotated Synthetic Datasets](https://mlsanity.com/paper/2506.08013/) (2025)
- [SeeClear: Reliable Transparent Object Depth Estimation via Generative Opacification](https://mlsanity.com/paper/2603.19547/) (2026)
- [Generative Perception of Shape and Material from Differential Motion](https://mlsanity.com/paper/2506.02473/) (2025)
- [From Gallery to Wrist: Realistic 3D Bracelet Insertion in Videos](https://mlsanity.com/paper/2507.20331/) (2025)
- [VT-Intrinsic: Physics-Based Decomposition of Reflectance and Shading using a Single Visible-Thermal Image Pair](https://mlsanity.com/paper/2509.10388/) (2025)
- [MVID: Feed-Forward Multi-View Intrinsic Image Decomposition](https://mlsanity.com/paper/2512.23667/) (2025)
- [FlowIID: Single-Step Intrinsic Image Decomposition via Latent Flow Matching](https://mlsanity.com/paper/2601.12329/) (2026)
- [Workflow-Aware Structured Layer Decomposition for Illustration Production](https://mlsanity.com/paper/2603.14925/) (2026)
- [SGS-Intrinsic: Semantic-Invariant Gaussian Splatting for Sparse-View Indoor Inverse Rendering](https://mlsanity.com/paper/2603.27516/) (2026)
- [Texture-aware Intrinsic Image Decomposition with Model- and Learning-based Priors](https://mlsanity.com/paper/2509.09352/) (2025)
- [IPCD: Intrinsic Point-Cloud Decomposition](https://mlsanity.com/paper/2511.09866/) (2025)
- [Clair Obscur: an Illumination-Aware Method for Real-World Image Vectorization](https://mlsanity.com/paper/2511.20034/) (2025)
- [LumiX: Structured and Coherent Text-to-Intrinsic Generation](https://mlsanity.com/paper/2512.02781/) (2025)

## Abstract (arXiv)

Intrinsic image decomposition aims to separate the surface reflectance and the effects from the illumination given a single photograph. Due to the complexity of the problem, most prior works assume a single-color illumination and a Lambertian world, which limits their use in illumination-aware image editing applications. In this work, we separate an input image into its diffuse albedo, colorful diffuse shading, and specular residual components. We arrive at our result by gradually removing first the single-color illumination and then the Lambertian-world assumptions. We show that by dividing the problem into easier sub-problems, in-the-wild colorful diffuse shading estimation can be achieved despite the limited ground-truth datasets. Our extended intrinsic model enables illumination-aware analysis of photographs and can be used for image editing applications such as specularity removal and per-pixel white balancing.