# Harmonizer: Learning to Perform White-Box Image and Video Harmonization

arXiv: https://arxiv.org/abs/2207.01322 · HTML: https://mlsanity.com/paper/2207.01322/

## Key facts

- Cited by 61 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 48 references that are in the map.
- First posted on arXiv 4 Jul 2022.

## Builds on (most cited first, 24 of 24)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [U2-Net: Going Deeper with Nested U-Structure for Salient Object Detection](https://mlsanity.com/paper/2005.09007/) (2020)
- [Learning Blind Video Temporal Consistency](https://mlsanity.com/paper/1808.00449/) (2018)
- [Robust High-Resolution Video Matting with Temporal Guidance](https://mlsanity.com/paper/2108.11515/) (2021)
- [DoveNet: Deep Image Harmonization via Domain Verification](https://mlsanity.com/paper/1911.13239/) (2019)
- [MODNet: Real-Time Trimap-Free Portrait Matting via Objective Decomposition](https://mlsanity.com/paper/2011.11961/) (2020)
- [Deep Image Harmonization](https://mlsanity.com/paper/1703.00069/) (2017)
- [Blind Video Temporal Consistency via Deep Video Prior](https://mlsanity.com/paper/2010.11838/) (2020)
- [Improving the Harmony of the Composite Image by Spatial-Separated Attention Module](https://mlsanity.com/paper/1907.06406/) (2019)
- [Region-aware Adaptive Instance Normalization for Image Harmonization](https://mlsanity.com/paper/2106.02853/) (2021)
- [SSH: A Self-Supervised Framework for Image Harmonization](https://mlsanity.com/paper/2108.06805/) (2021)
- [Deep White-Balance Editing](https://mlsanity.com/paper/2004.01354/) (2020)
- [Foreground-aware Semantic Representations for Image Harmonization](https://mlsanity.com/paper/2006.00809/) (2020)
- [Learning a Discriminative Model for the Perception of Realism in Composite Images](https://mlsanity.com/paper/1510.00477/) (2015)
- [BargainNet: Background-Guided Domain Translation for Image Harmonization](https://mlsanity.com/paper/2009.09169/) (2020)
- [Stylized Neural Painting](https://mlsanity.com/paper/2011.08114/) (2020)
- [Exposure: A White-Box Photo Post-Processing Framework](https://mlsanity.com/paper/1709.09602/) (2017)
- [Spatial-Separated Curve Rendering Network for Efficient and High-Resolution Image Harmonization](https://mlsanity.com/paper/2109.05750/) (2021)
- [Deep Painterly Harmonization](https://mlsanity.com/paper/1804.03189/) (2018)
- [Temporally Coherent Video Harmonization Using Adversarial Networks](https://mlsanity.com/paper/1809.01372/) (2018)

## Built on by (most cited first, 30 of 61)

- [IMPRINT: Generative Object Compositing by Learning Identity-Preserving Representation](https://mlsanity.com/paper/2403.10701/) (2024)
- [Relightful Harmonization: Lighting-aware Portrait Background Replacement](https://mlsanity.com/paper/2312.06886/) (2023)
- [ControlCom: Controllable Image Composition using Diffusion Model](https://mlsanity.com/paper/2308.10040/) (2023)
- [JarvisArt: Liberating Human Artistic Creativity via an Intelligent Photo Retouching Agent](https://mlsanity.com/paper/2506.17612/) (2025)
- [DreamEdit: Subject-driven Image Editing](https://mlsanity.com/paper/2306.12624/) (2023)
- [Neural Preset for Color Style Transfer](https://mlsanity.com/paper/2303.13511/) (2023)
- [LBM: Latent Bridge Matching for Fast Image-to-Image Translation](https://mlsanity.com/paper/2503.07535/) (2025)
- [Semi-supervised Parametric Real-world Image Harmonization](https://mlsanity.com/paper/2303.00157/) (2023)
- [MatAnyone: Stable Video Matting with Consistent Memory Propagation](https://mlsanity.com/paper/2501.14677/) (2025)
- [Dense Pixel-to-Pixel Harmonization via Continuous Image Representation](https://mlsanity.com/paper/2303.01681/) (2023)
- [LayerFusion: Harmonized Multi-Layer Text-to-Image Generation with Generative Priors](https://mlsanity.com/paper/2412.04460/) (2024)
- [DreamCom: Finetuning Text-guided Inpainting Model for Image Composition](https://mlsanity.com/paper/2309.15508/) (2023)
- [Geometry Transfer for Stylizing Radiance Fields](https://mlsanity.com/paper/2402.00863/) (2024)
- [RSFNet: A White-Box Image Retouching Approach using Region-Specific Color Filters](https://mlsanity.com/paper/2303.08682/) (2023)
- [DiFaReli++: Diffusion Face Relighting with Consistent Cast Shadows](https://mlsanity.com/paper/2304.09479/) (2023)
- [Hierarchical Dynamic Image Harmonization](https://mlsanity.com/paper/2211.08639/) (2022)
- [LEMaRT: Label-Efficient Masked Region Transform for Image Harmonization](https://mlsanity.com/paper/2304.13166/) (2023)
- [Deep Image Harmonization in Dual Color Spaces](https://mlsanity.com/paper/2308.02813/) (2023)
- [Intrinsic Harmonization for Illumination-Aware Compositing](https://mlsanity.com/paper/2312.03698/) (2023)
- [ORIDa: Object-centric Real-world Image Composition Dataset](https://mlsanity.com/paper/2506.08964/) (2025)
- [DiffusionHarmonizer: Bridging Neural Reconstruction and Photorealistic Simulation with Online Diffusion Enhancer](https://mlsanity.com/paper/2602.24096/) (2026)
- [PhotoArtAgent: Intelligent Photo Retouching with Language Model-Based Artist Agents](https://mlsanity.com/paper/2505.23130/) (2025)
- [Zero-Shot Image Harmonization with Generative Model Prior](https://mlsanity.com/paper/2307.08182/) (2023)
- [MVOC: a training-free multiple video object composition method with diffusion models](https://mlsanity.com/paper/2406.15829/) (2024)
- [DreamLight: Towards Harmonious and Consistent Image Relighting](https://mlsanity.com/paper/2506.14549/) (2025)
- [GenHSI: Controllable Generation of Human-Scene Interaction Videos](https://mlsanity.com/paper/2506.19840/) (2025)
- [Deep Image Harmonization with Learnable Augmentation](https://mlsanity.com/paper/2308.00376/) (2023)
- [HandEdit: A Unified Benchmark for Egocentric Human-to-Robot Dexterous Hand Image Editing](https://mlsanity.com/paper/2608.12122/) (2026)
- [Learning Global-aware Kernel for Image Harmonization](https://mlsanity.com/paper/2305.11676/) (2023)
- [Deep Image Harmonization with Globally Guided Feature Transformation and Relation Distillation](https://mlsanity.com/paper/2308.00356/) (2023)

## Abstract (arXiv)

Recent works on image harmonization solve the problem as a pixel-wise image translation task via large autoencoders. They have unsatisfactory performances and slow inference speeds when dealing with high-resolution images. In this work, we observe that adjusting the input arguments of basic image filters, e.g., brightness and contrast, is sufficient for humans to produce realistic images from the composite ones. Hence, we frame image harmonization as an image-level regression problem to learn the arguments of the filters that humans use for the task. We present a Harmonizer framework for image harmonization. Unlike prior methods that are based on black-box autoencoders, Harmonizer contains a neural network for filter argument prediction and several white-box filters (based on the predicted arguments) for image harmonization. We also introduce a cascade regressor and a dynamic loss strategy for Harmonizer to learn filter arguments more stably and precisely. Since our network only outputs image-level arguments and the filters we used are efficient, Harmonizer is much lighter and faster than existing methods. Comprehensive experiments demonstrate that Harmonizer surpasses existing methods notably, especially with high-resolution inputs. Finally, we apply Harmonizer to video harmonization, which achieves consistent results across frames and 56 fps at 1080P resolution. Code and models are available at: https://github.com/ZHKKKe/Harmonizer.