# Comprehensive Relighting: Generalizable and Consistent Monocular Human Relighting and Harmonization

arXiv: https://arxiv.org/abs/2504.03011 · HTML: https://mlsanity.com/paper/2504.03011/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models.
- Builds on 49 of its 74 references that are in the map.
- First posted on arXiv 3 Apr 2025.

## Placed in

- Diffusion models / [3D, point cloud and scene generation](https://mlsanity.com/diffusion/3d-point-cloud-and-scene-generation/)

## Builds on (most cited first, 30 of 49)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [One-step Diffusion with Distribution Matching Distillation](https://mlsanity.com/paper/2311.18828/) (2023)
- [Residual Dense Network for Image Super-Resolution](https://mlsanity.com/paper/1802.08797/) (2018)
- [PVT v2: Improved Baselines with Pyramid Vision Transformer](https://mlsanity.com/paper/2106.13797/) (2021)
- [TokenFlow: Consistent Diffusion Features for Consistent Video Editing](https://mlsanity.com/paper/2307.10373/) (2023)
- [Liquid Warping GAN: A Unified Framework for Human Motion Imitation, Appearance Transfer and Novel View Synthesis](https://mlsanity.com/paper/1909.12224/) (2019)
- [Single Image Portrait Relighting](https://mlsanity.com/paper/1905.00824/) (2019)
- [Editable Free-viewpoint Video Using a Layered Neural Representation](https://mlsanity.com/paper/2104.14786/) (2021)
- [DoveNet: Deep Image Harmonization via Domain Verification](https://mlsanity.com/paper/1911.13239/) (2019)
- [Deep Image Harmonization](https://mlsanity.com/paper/1703.00069/) (2017)
- [SwitchLight: Co-design of Physics-driven Architecture and Pre-training Framework for Human Portrait Relighting](https://mlsanity.com/paper/2402.18848/) (2024)
- [Deep 3D Capture: Geometry and Reflectance from Sparse Multi-View Images](https://mlsanity.com/paper/2003.12642/) (2020)
- [SSH: A Self-Supervised Framework for Image Harmonization](https://mlsanity.com/paper/2108.06805/) (2021)
- [Relightful Harmonization: Lighting-aware Portrait Background Replacement](https://mlsanity.com/paper/2312.06886/) (2023)
- [High-Resolution Image Harmonization via Collaborative Dual Transformations](https://mlsanity.com/paper/2109.06671/) (2021)
- [Learning a Discriminative Model for the Perception of Realism in Composite Images](https://mlsanity.com/paper/1510.00477/) (2015)
- [Neural Video Portrait Relighting in Real-time via Consistency Modeling](https://mlsanity.com/paper/2104.00484/) (2021)
- [Face Relighting with Geometrically Consistent Shadows](https://mlsanity.com/paper/2203.16681/) (2022)
- [Towards High Fidelity Face Relighting with Realistic Shadows](https://mlsanity.com/paper/2104.00825/) (2021)

## Built on by (most cited first, 12 of 12)

- [Lumen: Consistent Video Relighting and Harmonious Background Replacement with Video Generative Models](https://mlsanity.com/paper/2508.12945/) (2025)
- [LuxRemix: Lighting Decomposition and Remixing for Indoor Scenes](https://mlsanity.com/paper/2601.15283/) (2026)
- [Light4D: Training-Free Extreme Viewpoint 4D Video Relighting](https://mlsanity.com/paper/2602.11769/) (2026)
- [ReLumix: Extending Image Relighting to Video via Video Diffusion Models](https://mlsanity.com/paper/2509.23769/) (2025)
- [TokenLight: Precise Lighting Control in Images using Attribute Tokens](https://mlsanity.com/paper/2604.15310/) (2026)
- [FashionPose: Unified Text-Driven Fashion Synthesis with Joint Geometric and Photometric Control](https://mlsanity.com/paper/2507.13311/) (2025)
- [HeadLighter: Disentangling Illumination in Generative 3D Gaussian Heads via Lightstage Captures](https://mlsanity.com/paper/2601.02103/) (2026)
- [Lighting-aware Unified Model for Instance Segmentation](https://mlsanity.com/paper/2605.20436/) (2026)
- [BodyReLux: Temporally Consistent Full-Body Video Relighting](https://mlsanity.com/paper/2605.21766/) (2026)
- [Pixel Cube: Diffusion-based Portrait Video Relighting Through Realistic Lighting Reproduction](https://mlsanity.com/paper/2606.02919/) (2026)
- [Generative Relightable Avatars](https://mlsanity.com/paper/2606.22718/) (2026)
- [Consistent Feature Transport for Image Relighting](https://mlsanity.com/paper/2607.17833/) (2026)

## Abstract (arXiv)

This paper introduces Comprehensive Relighting, the first all-in-one approach that can both control and harmonize the lighting from an image or video of humans with arbitrary body parts from any scene. Building such a generalizable model is extremely challenging due to the lack of dataset, restricting existing image-based relighting models to a specific scenario (e.g., face or static human). To address this challenge, we repurpose a pre-trained diffusion model as a general image prior and jointly model the human relighting and background harmonization in the coarse-to-fine framework. To further enhance the temporal coherence of the relighting, we introduce an unsupervised temporal lighting model that learns the lighting cycle consistency from many real-world videos without any ground truth. In inference time, our temporal lighting module is combined with the diffusion models through the spatio-temporal feature blending algorithms without extra training; and we apply a new guided refinement as a post-processing to preserve the high-frequency details from the input image. In the experiments, Comprehensive Relighting shows a strong generalizability and lighting temporal coherence, outperforming existing image-based human relighting and harmonization methods.