# SIDNet: Learning Shading-aware Illumination Descriptor for Image Harmonization

arXiv: https://arxiv.org/abs/2112.01314 · HTML: https://mlsanity.com/paper/2112.01314/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 57 references that are in the map.
- First posted on arXiv 2 Dec 2021.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Residual Dense Network for Image Super-Resolution](https://mlsanity.com/paper/1802.08797/) (2018)
- [Single Image Portrait Relighting](https://mlsanity.com/paper/1905.00824/) (2019)
- [DoveNet: Deep Image Harmonization via Domain Verification](https://mlsanity.com/paper/1911.13239/) (2019)
- [Deep Image Harmonization](https://mlsanity.com/paper/1703.00069/) (2017)
- [Improving the Harmony of the Composite Image by Spatial-Separated Attention Module](https://mlsanity.com/paper/1907.06406/) (2019)
- [Region-aware Adaptive Instance Normalization for Image Harmonization](https://mlsanity.com/paper/2106.02853/) (2021)
- [SSH: A Self-Supervised Framework for Image Harmonization](https://mlsanity.com/paper/2108.06805/) (2021)
- [High-Resolution Image Harmonization via Collaborative Dual Transformations](https://mlsanity.com/paper/2109.06671/) (2021)
- [Foreground-aware Semantic Representations for Image Harmonization](https://mlsanity.com/paper/2006.00809/) (2020)
- [Learning a Discriminative Model for the Perception of Realism in Composite Images](https://mlsanity.com/paper/1510.00477/) (2015)
- [BargainNet: Background-Guided Domain Translation for Image Harmonization](https://mlsanity.com/paper/2009.09169/) (2020)
- [Self-supervised Outdoor Scene Relighting](https://mlsanity.com/paper/2107.03106/) (2021)
- [Single-image Full-body Human Relighting](https://mlsanity.com/paper/2107.07259/) (2021)
- [Deep Image Harmonization by Bridging the Reality Gap](https://mlsanity.com/paper/2103.17104/) (2021)
- [Deep Image-based Illumination Harmonization](https://mlsanity.com/paper/2108.00150/) (2021)

## Built on by (most cited first, 2 of 2)

- [Efficient HDR Reconstruction from Real-World Raw Images](https://mlsanity.com/paper/2306.10311/) (2023)
- [Region-to-Region: Enhancing Generative Image Harmonization with Adaptive Regional Injection](https://mlsanity.com/paper/2508.09746/) (2025)

## Abstract (arXiv)

Image harmonization aims at adjusting the appearance of the foreground to make it more compatible with the background. Without exploring background illumination and its effects on the foreground elements, existing works are incapable of generating a realistic foreground shading. In this paper, we decompose the image harmonization task into two sub-problems: 1) illumination estimation of the background image and 2) re-rendering of foreground objects under background illumination. Before solving these two sub-problems, we first learn a shading-aware illumination descriptor via a well-designed neural rendering framework, of which the key is a shading bases module that generates multiple shading bases from the foreground image. Then we design a background illumination estimation module to extract the illumination descriptor from the background. Finally, the Shading-aware Illumination Descriptor is used in conjunction with the neural rendering framework (SIDNet) to produce the harmonized foreground image containing a novel harmonized shading. Moreover, we construct a photo-realistic synthetic image harmonization dataset that contains numerous shading variations with image-based lighting. Extensive experiments on both synthetic and real data demonstrate the superiority of the proposed method, especially in dealing with foreground shadings.