# Deep Image-based Illumination Harmonization arXiv: https://arxiv.org/abs/2108.00150 ยท HTML: https://mlsanity.com/paper/2108.00150/ ## Key facts - Cited by 11 papers in the map (data through 14 Oct 2026). - Builds on 25 of its 44 references that are in the map. - First posted on arXiv 31 Jul 2021. ## Builds on (most cited first, 25 of 25) - [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015) - [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015) - [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014) - [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016) - [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016) - [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014) - [Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network](https://mlsanity.com/paper/1609.04802/) (2016) - [Instance Normalization: The Missing Ingredient for Fast Stylization](https://mlsanity.com/paper/1607.08022/) (2016) - [Accurate Image Super-Resolution Using Very Deep Convolutional Networks](https://mlsanity.com/paper/1511.04587/) (2015) - [Wasserstein GAN](https://mlsanity.com/paper/1701.07875/) (2017) - [Learning to Predict Indoor Illumination from a Single Image](https://mlsanity.com/paper/1704.00090/) (2017) - [Stacked Conditional Generative Adversarial Networks for Jointly Learning Shadow Detection and Shadow Removal](https://mlsanity.com/paper/1712.02478/) (2017) - [DoveNet: Deep Image Harmonization via Domain Verification](https://mlsanity.com/paper/1911.13239/) (2019) - [InverseRenderNet: Learning single image inverse rendering](https://mlsanity.com/paper/1811.12328/) (2018) - [Mask-ShadowGAN: Learning to Remove Shadows from Unpaired Data](https://mlsanity.com/paper/1903.10683/) (2019) - [Deep Parametric Indoor Lighting Estimation](https://mlsanity.com/paper/1910.08812/) (2019) - [Rendering Synthetic Objects into Legacy Photographs](https://mlsanity.com/paper/1912.11565/) (2019) - [Region-aware Adaptive Instance Normalization for Image Harmonization](https://mlsanity.com/paper/2106.02853/) (2021) - [An Empirical Study of Spatial Attention Mechanisms in Deep Networks](https://mlsanity.com/paper/1904.05873/) (2019) - [ARGAN: Attentive Recurrent Generative Adversarial Network for Shadow Detection and Removal](https://mlsanity.com/paper/1908.01323/) (2019) - [A Hybrid Attention Mechanism for Weakly-Supervised Temporal Action Localization](https://mlsanity.com/paper/2101.00545/) (2021) - [Automatic Scene Inference for 3D Object Compositing](https://mlsanity.com/paper/1912.12297/) (2019) - [Adversarial Image Composition with Auxiliary Illumination](https://mlsanity.com/paper/2009.08255/) (2020) - [An Approximate Shading Model with Detail Decomposition for Object Relighting](https://mlsanity.com/paper/1804.07514/) (2018) - [Iterative and Adaptive Sampling with Spatial Attention for Black-Box Model Explanations](https://mlsanity.com/paper/1912.08387/) (2019) ## Built on by (most cited first, 11 of 11) - [ControlCom: Controllable Image Composition using Diffusion Model](https://mlsanity.com/paper/2308.10040/) (2023) - [LightSim: Neural Lighting Simulation for Urban Scenes](https://mlsanity.com/paper/2312.06654/) (2023) - [DreamCom: Finetuning Text-guided Inpainting Model for Image Composition](https://mlsanity.com/paper/2309.15508/) (2023) - [Deep Image Harmonization in Dual Color Spaces](https://mlsanity.com/paper/2308.02813/) (2023) - [HarmonyIQA: Pioneering Benchmark and Model for Image Harmonization Quality Assessment](https://mlsanity.com/paper/2501.01116/) (2025) - [Deep Image Harmonization with Globally Guided Feature Transformation and Relation Distillation](https://mlsanity.com/paper/2308.00356/) (2023) - [SIDNet: Learning Shading-aware Illumination Descriptor for Image Harmonization](https://mlsanity.com/paper/2112.01314/) (2021) - [Inharmonious Region Localization with Auxiliary Style Feature](https://mlsanity.com/paper/2210.02029/) (2022) - [Diverse Image Harmonization](https://mlsanity.com/paper/2407.15481/) (2024) - [From Gallery to Wrist: Realistic 3D Bracelet Insertion in Videos](https://mlsanity.com/paper/2507.20331/) (2025) - [Inharmonious Region Localization via Recurrent Self-Reasoning](https://mlsanity.com/paper/2210.02036/) (2022) ## Abstract (arXiv) Integrating a foreground object into a background scene with illumination harmonization is an important but challenging task in computer vision and augmented reality community. Existing methods mainly focus on foreground and background appearance consistency or the foreground object shadow generation, which rarely consider global appearance and illumination harmonization. In this paper, we formulate seamless illumination harmonization as an illumination exchange and aggregation problem. Specifically, we firstly apply a physically-based rendering method to construct a large-scale, high-quality dataset (named IH) for our task, which contains various types of foreground objects and background scenes with different lighting conditions. Then, we propose a deep image-based illumination harmonization GAN framework named DIH-GAN, which makes full use of a multi-scale attention mechanism and illumination exchange strategy to directly infer mapping relationship between the inserted foreground object and the corresponding background scene. Meanwhile, we also use adversarial learning strategy to further refine the illumination harmonization result. Our method can not only achieve harmonious appearance and illumination for the foreground object but also can generate compelling shadow cast by the foreground object. Comprehensive experiments on both our IH dataset and real-world images show that our proposed DIH-GAN provides a practical and effective solution for image-based object illumination harmonization editing, and validate the superiority of our method against state-of-the-art methods. Our IH dataset is available at https://github.com/zhongyunbao/Dataset.