# Toward Interactive Modulation for Photo-Realistic Image Restoration

arXiv: https://arxiv.org/abs/2105.03085 · HTML: https://mlsanity.com/paper/2105.03085/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 40 references that are in the map.
- First posted on arXiv 7 May 2021.

## Builds on (most cited first, 30 of 32)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016)
- [Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network](https://mlsanity.com/paper/1609.04802/) (2016)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising](https://mlsanity.com/paper/1608.03981/) (2016)
- [ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks](https://mlsanity.com/paper/1809.00219/) (2018)
- [Image Quality Assessment: Unifying Structure and Texture Similarity](https://mlsanity.com/paper/2004.07728/) (2020)
- [Empirical Evaluation of Rectified Activations in Convolutional Network](https://mlsanity.com/paper/1505.00853/) (2015)
- [FFDNet: Toward a Fast and Flexible Solution for CNN based Image Denoising](https://mlsanity.com/paper/1710.04026/) (2017)
- [Learning Deep CNN Denoiser Prior for Image Restoration](https://mlsanity.com/paper/1704.03264/) (2017)
- [Toward Convolutional Blind Denoising of Real Photographs](https://mlsanity.com/paper/1807.04686/) (2018)
- [SRFlow: Learning the Super-Resolution Space with Normalizing Flow](https://mlsanity.com/paper/2006.14200/) (2020)
- [Blind Super-Resolution With Iterative Kernel Correction](https://mlsanity.com/paper/1904.03377/) (2019)
- [Trainable Nonlinear Reaction Diffusion: A Flexible Framework for Fast and Effective Image Restoration](https://mlsanity.com/paper/1508.02848/) (2015)
- [PIPAL: a Large-Scale Image Quality Assessment Dataset for Perceptual Image Restoration](https://mlsanity.com/paper/2007.12142/) (2020)
- [DSLR-Quality Photos on Mobile Devices with Deep Convolutional Networks](https://mlsanity.com/paper/1704.02470/) (2017)
- [The 2018 PIRM Challenge on Perceptual Image Super-resolution](https://mlsanity.com/paper/1809.07517/) (2018)
- [RankSRGAN: Generative Adversarial Networks with Ranker for Image Super-Resolution](https://mlsanity.com/paper/1908.06382/) (2019)
- [Compression Artifacts Reduction by a Deep Convolutional Network](https://mlsanity.com/paper/1504.06993/) (2015)
- [Frequency Separation for Real-World Super-Resolution](https://mlsanity.com/paper/1911.07850/) (2019)
- [Deep Plug-and-Play Super-Resolution for Arbitrary Blur Kernels](https://mlsanity.com/paper/1903.12529/) (2019)
- [NTIRE 2020 Challenge on Real-World Image Super-Resolution: Methods and Results](https://mlsanity.com/paper/2005.01996/) (2020)
- [Conditional Sequential Modulation for Efficient Global Image Retouching](https://mlsanity.com/paper/2009.10390/) (2020)
- [Deep Network Interpolation for Continuous Imagery Effect Transition](https://mlsanity.com/paper/1811.10515/) (2018)
- [Explorable Super Resolution](https://mlsanity.com/paper/1912.01839/) (2019)
- [Modulating Image Restoration with Continual Levels via Adaptive Feature Modification Layers](https://mlsanity.com/paper/1904.08118/) (2019)
- [AIM 2019 Challenge on Real-World Image Super-Resolution: Methods and Results](https://mlsanity.com/paper/1911.07783/) (2019)
- [CFSNet: Toward a Controllable Feature Space for Image Restoration](https://mlsanity.com/paper/1904.00634/) (2019)
- [Natural Image Noise Dataset](https://mlsanity.com/paper/1906.00270/) (2019)

## Built on by (most cited first, 15 of 15)

- [Adversarial Diffusion Compression for Real-World Image Super-Resolution](https://mlsanity.com/paper/2411.13383/) (2024)
- [NTIRE 2022 Challenge on Perceptual Image Quality Assessment](https://mlsanity.com/paper/2206.11695/) (2022)
- [MM-RealSR: Metric Learning based Interactive Modulation for Real-World Super-Resolution](https://mlsanity.com/paper/2205.05065/) (2022)
- [Dynamic Path-Controllable Deep Unfolding Network for Compressive Sensing](https://mlsanity.com/paper/2306.16060/) (2023)
- [Real-time Controllable Denoising for Image and Video](https://mlsanity.com/paper/2303.16425/) (2023)
- [Self-Supervised Scalable Deep Compressed Sensing](https://mlsanity.com/paper/2308.13777/) (2023)
- [Restore Anything Pipeline: Segment Anything Meets Image Restoration](https://mlsanity.com/paper/2305.13093/) (2023)
- [On the Importance of Denoising when Learning to Compress Images](https://mlsanity.com/paper/2307.06233/) (2023)
- [Tunable Convolutions with Parametric Multi-Loss Optimization](https://mlsanity.com/paper/2304.00898/) (2023)
- [Contrastive Monotonic Pixel-Level Modulation](https://mlsanity.com/paper/2207.11517/) (2022)
- [IRConStyle: Image Restoration Framework Using Contrastive Learning and Style Transfer](https://mlsanity.com/paper/2402.15784/) (2024)
- [Enhancing Quality of Compressed Images by Mitigating Enhancement Bias Towards Compression Domain](https://mlsanity.com/paper/2402.17200/) (2024)
- [QueryCDR: Query-Based Controllable Distortion Rectification Network for Fisheye Images](https://mlsanity.com/paper/2412.13496/) (2024)
- [Continuous Facial Motion Deblurring](https://mlsanity.com/paper/2207.06626/) (2022)
- [WaterGen: Decoupling Scene and Medium in Underwater Image Generation](https://mlsanity.com/paper/2606.31147/) (2026)

## Abstract (arXiv)

Modulating image restoration level aims to generate a restored image by altering a factor that represents the restoration strength. Previous works mainly focused on optimizing the mean squared reconstruction error, which brings high reconstruction accuracy but lacks finer texture details. This paper presents a Controllable Unet Generative Adversarial Network (CUGAN) to generate high-frequency textures in the modulation tasks. CUGAN consists of two modules -- base networks and condition networks. The base networks comprise a generator and a discriminator. In the generator, we realize the interactive control of restoration levels by tuning the weights of different features from different scales in the Unet architecture. Moreover, we adaptively modulate the intermediate features in the discriminator according to the severity of degradations. The condition networks accept the condition vector (encoded degradation information) as input, then generate modulation parameters for both the generator and the discriminator. During testing, users can control the output effects by tweaking the condition vector. We also provide a smooth transition between GAN and MSE effects by a simple transition method. Extensive experiments demonstrate that the proposed CUGAN achieves excellent performance on image restoration modulation tasks.