# Spatial-Separated Curve Rendering Network for Efficient and High-Resolution Image Harmonization

arXiv: https://arxiv.org/abs/2109.05750 · HTML: https://mlsanity.com/paper/2109.05750/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 51 references that are in the map.
- First posted on arXiv 13 Sep 2021.

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [Semantic Understanding of Scenes through the ADE20K Dataset](https://mlsanity.com/paper/1608.05442/) (2016)
- [SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size](https://mlsanity.com/paper/1602.07360/) (2016)
- [Least Squares Generative Adversarial Networks](https://mlsanity.com/paper/1611.04076/) (2016)
- [Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks](https://mlsanity.com/paper/1703.10593/) (2017)
- [Zero-Reference Deep Curve Estimation for Low-Light Image Enhancement](https://mlsanity.com/paper/2001.06826/) (2020)
- [Deep Bilateral Learning for Real-Time Image Enhancement](https://mlsanity.com/paper/1707.02880/) (2017)
- [Learning Image-adaptive 3D Lookup Tables for High Performance Photo Enhancement in Real-time](https://mlsanity.com/paper/2009.14468/) (2020)
- [Learning Rich Features for Image Manipulation Detection](https://mlsanity.com/paper/1805.04953/) (2018)
- [DoveNet: Deep Image Harmonization via Domain Verification](https://mlsanity.com/paper/1911.13239/) (2019)
- [Deep Image Harmonization](https://mlsanity.com/paper/1703.00069/) (2017)
- [Improving the Harmony of the Composite Image by Spatial-Separated Attention Module](https://mlsanity.com/paper/1907.06406/) (2019)
- [Region-aware Adaptive Instance Normalization for Image Harmonization](https://mlsanity.com/paper/2106.02853/) (2021)
- [SSH: A Self-Supervised Framework for Image Harmonization](https://mlsanity.com/paper/2108.06805/) (2021)
- [Foreground-aware Semantic Representations for Image Harmonization](https://mlsanity.com/paper/2006.00809/) (2020)
- [Learning a Discriminative Model for the Perception of Realism in Composite Images](https://mlsanity.com/paper/1510.00477/) (2015)
- [BargainNet: Background-Guided Domain Translation for Image Harmonization](https://mlsanity.com/paper/2009.09169/) (2020)
- [Cascading Convolutional Color Constancy](https://mlsanity.com/paper/1912.11180/) (2019)
- [Split then Refine: Stacked Attention-guided ResUNets for Blind Single Image Visible Watermark Removal](https://mlsanity.com/paper/2012.07007/) (2020)
- [Blind Visual Motif Removal from a Single Image](https://mlsanity.com/paper/1904.02756/) (2019)
- [Inserting Videos into Videos](https://mlsanity.com/paper/1903.06571/) (2019)

## Built on by (most cited first, 25 of 25)

- [Harmonizer: Learning to Perform White-Box Image and Video Harmonization](https://mlsanity.com/paper/2207.01322/) (2022)
- [Relightful Harmonization: Lighting-aware Portrait Background Replacement](https://mlsanity.com/paper/2312.06886/) (2023)
- [ControlCom: Controllable Image Composition using Diffusion Model](https://mlsanity.com/paper/2308.10040/) (2023)
- [High-Resolution Document Shadow Removal via A Large-Scale Real-World Dataset and A Frequency-Aware Shadow Erasing Net](https://mlsanity.com/paper/2308.14221/) (2023)
- [CoordFill: Efficient High-Resolution Image Inpainting via Parameterized Coordinate Querying](https://mlsanity.com/paper/2303.08524/) (2023)
- [Semi-supervised Parametric Real-world Image Harmonization](https://mlsanity.com/paper/2303.00157/) (2023)
- [Dense Pixel-to-Pixel Harmonization via Continuous Image Representation](https://mlsanity.com/paper/2303.01681/) (2023)
- [DreamCom: Finetuning Text-guided Inpainting Model for Image Composition](https://mlsanity.com/paper/2309.15508/) (2023)
- [ShaDocNet: Learning Spatial-Aware Tokens in Transformer for Document Shadow Removal](https://mlsanity.com/paper/2211.16675/) (2022)
- [Hierarchical Dynamic Image Harmonization](https://mlsanity.com/paper/2211.08639/) (2022)
- [LEMaRT: Label-Efficient Masked Region Transform for Image Harmonization](https://mlsanity.com/paper/2304.13166/) (2023)
- [Painterly Image Harmonization using Diffusion Model](https://mlsanity.com/paper/2308.02228/) (2023)
- [Deep Image Harmonization in Dual Color Spaces](https://mlsanity.com/paper/2308.02813/) (2023)
- [FRIH: Fine-grained Region-aware Image Harmonization](https://mlsanity.com/paper/2205.06448/) (2022)
- [Zero-Shot Image Harmonization with Generative Model Prior](https://mlsanity.com/paper/2307.08182/) (2023)
- [Deep Image Harmonization with Learnable Augmentation](https://mlsanity.com/paper/2308.00376/) (2023)
- [Deep Image Harmonization with Globally Guided Feature Transformation and Relation Distillation](https://mlsanity.com/paper/2308.00356/) (2023)
- [Painterly Image Harmonization by Learning from Painterly Objects](https://mlsanity.com/paper/2312.10263/) (2023)
- [Image Harmonization with Region-wise Contrastive Learning](https://mlsanity.com/paper/2205.14058/) (2022)
- [Progressive Painterly Image Harmonization from Low-level Styles to High-level Styles](https://mlsanity.com/paper/2312.10264/) (2023)
- [Segment Anything Model Meets Image Harmonization](https://mlsanity.com/paper/2312.12729/) (2023)
- [Diverse Image Harmonization](https://mlsanity.com/paper/2407.15481/) (2024)
- [Painterly Image Harmonization via Adversarial Residual Learning](https://mlsanity.com/paper/2311.08646/) (2023)
- [DepGAN: Leveraging Depth Maps for Handling Occlusions and Transparency in Image Composition](https://mlsanity.com/paper/2407.11890/) (2024)
- [Retrieval Augmented Image Harmonization](https://mlsanity.com/paper/2412.13916/) (2024)

## Abstract (arXiv)

Image harmonization aims to modify the color of the composited region with respect to the specific background. Previous works model this task as a pixel-wise image-to-image translation using UNet family structures. However, the model size and computational cost limit the ability of their models on edge devices and higher-resolution images. To this end, we propose a novel spatial-separated curve rendering network(S2CRNet) for efficient and high-resolution image harmonization for the first time. In S2CRNet, we firstly extract the spatial-separated embeddings from the thumbnails of the masked foreground and background individually. Then, we design a curve rendering module(CRM), which learns and combines the spatial-specific knowledge using linear layers to generate the parameters of the piece-wise curve mapping in the foreground region. Finally, we directly render the original high-resolution images using the learned color curve. Besides, we also make two extensions of the proposed framework via the Cascaded-CRM and Semantic-CRM for cascaded refinement and semantic guidance, respectively. Experiments show that the proposed method reduces more than 90% parameters compared with previous methods but still achieves the state-of-the-art performance on both synthesized iHarmony4 and real-world DIH test sets. Moreover, our method can work smoothly on higher resolution images(eg., 2048×2048) in 0.1 seconds with much lower GPU computational resources than all existing methods. The code will be made available at urlhttp://github.com/stefanLeong/S2CRNet.