# Real-Time Global Illumination Decomposition of Videos

arXiv: https://arxiv.org/abs/1908.01961 · HTML: https://mlsanity.com/paper/1908.01961/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Aug 2019.

## Built on by (most cited first, 18 of 18)

- [Physically-Based Editing of Indoor Scene Lighting from a Single Image](https://mlsanity.com/paper/2205.09343/) (2022)
- [IntrinsicNeRF: Learning Intrinsic Neural Radiance Fields for Editable Novel View Synthesis](https://mlsanity.com/paper/2210.00647/) (2022)
- [PaletteNeRF: Palette-based Appearance Editing of Neural Radiance Fields](https://mlsanity.com/paper/2212.10699/) (2022)
- [Colorful Diffuse Intrinsic Image Decomposition in the Wild](https://mlsanity.com/paper/2409.13690/) (2024)
- [Relighting4D: Neural Relightable Human from Videos](https://mlsanity.com/paper/2207.07104/) (2022)
- [OutCast: Outdoor Single-image Relighting with Cast Shadows](https://mlsanity.com/paper/2204.09341/) (2022)
- [Recasting Regional Lighting for Shadow Removal](https://mlsanity.com/paper/2402.00341/) (2024)
- [SIRe-IR: Inverse Rendering for BRDF Reconstruction with Shadow and Illumination Removal in High-Illuminance Scenes](https://mlsanity.com/paper/2310.13030/) (2023)
- [Layered Image Vectorization via Semantic Simplification](https://mlsanity.com/paper/2406.05404/) (2024)
- [All-photon Polarimetric Time-of-Flight Imaging](https://mlsanity.com/paper/2112.09278/) (2021)
- [Generative Portrait Shadow Removal](https://mlsanity.com/paper/2410.05525/) (2024)
- [LuxRemix: Lighting Decomposition and Remixing for Indoor Scenes](https://mlsanity.com/paper/2601.15283/) (2026)
- [IBL-NeRF: Image-Based Lighting Formulation of Neural Radiance Fields](https://mlsanity.com/paper/2210.08202/) (2022)
- [JoIN: Joint GANs Inversion for Intrinsic Image Decomposition](https://mlsanity.com/paper/2305.11321/) (2023)
- [LightCity: An Urban Dataset for Outdoor Inverse Rendering and Reconstruction under Multi-illumination Conditions](https://mlsanity.com/paper/2602.01118/) (2026)
- [MixLight: Borrowing the Best of both Spherical Harmonics and Gaussian Models](https://mlsanity.com/paper/2404.12768/) (2024)
- [ICTPolarReal: A Polarized Reflection and Material Dataset of Real World Objects](https://mlsanity.com/paper/2603.24912/) (2026)
- [Clair Obscur: an Illumination-Aware Method for Real-World Image Vectorization](https://mlsanity.com/paper/2511.20034/) (2025)

## Abstract (arXiv)

We propose the first approach for the decomposition of a monocular color video into direct and indirect illumination components in real time. We retrieve, in separate layers, the contribution made to the scene appearance by the scene reflectance, the light sources and the reflections from various coherent scene regions to one another. Existing techniques that invert global light transport require image capture under multiplexed controlled lighting, or only enable the decomposition of a single image at slow off-line frame rates. In contrast, our approach works for regular videos and produces temporally coherent decomposition layers at real-time frame rates. At the core of our approach are several sparsity priors that enable the estimation of the per-pixel direct and indirect illumination layers based on a small set of jointly estimated base reflectance colors. The resulting variational decomposition problem uses a new formulation based on sparse and dense sets of non-linear equations that we solve efficiently using a novel alternating data-parallel optimization strategy. We evaluate our approach qualitatively and quantitatively, and show improvements over the state of the art in this field, in both quality and runtime. In addition, we demonstrate various real-time appearance editing applications for videos with consistent illumination.