# Universal Face Restoration With Memorized Modulation

arXiv: https://arxiv.org/abs/2110.01033 · HTML: https://mlsanity.com/paper/2110.01033/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 55 references that are in the map.
- First posted on arXiv 3 Oct 2021.

## Builds on (most cited first, 30 of 40)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising](https://mlsanity.com/paper/1608.03981/) (2016)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [Demystifying MMD GANs](https://mlsanity.com/paper/1801.01401/) (2018)
- [Image Super-Resolution Using Very Deep Residual Channel Attention Networks](https://mlsanity.com/paper/1807.02758/) (2018)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [VGGFace2: A dataset for recognising faces across pose and age](https://mlsanity.com/paper/1710.08092/) (2017)
- [How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)](https://mlsanity.com/paper/1703.07332/) (2017)
- [DeblurGAN: Blind Motion Deblurring Using Conditional Adversarial Networks](https://mlsanity.com/paper/1711.07064/) (2017)
- [Memory Networks](https://mlsanity.com/paper/1410.3916/) (2014)
- [PULSE: Self-Supervised Photo Upsampling via Latent Space Exploration of Generative Models](https://mlsanity.com/paper/2003.03808/) (2020)
- [Towards Real-World Blind Face Restoration with Generative Facial Prior](https://mlsanity.com/paper/2101.04061/) (2021)
- [DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesis](https://mlsanity.com/paper/1904.01310/) (2019)
- [U-GAT-IT: Unsupervised Generative Attentional Networks with Adaptive Layer-Instance Normalization for Image-to-Image Translation](https://mlsanity.com/paper/1907.10830/) (2019)
- [Multi-level Wavelet-CNN for Image Restoration](https://mlsanity.com/paper/1805.07071/) (2018)
- [Non-Local Recurrent Network for Image Restoration](https://mlsanity.com/paper/1806.02919/) (2018)
- [GLEAN: Generative Latent Bank for Large-Factor Image Super-Resolution](https://mlsanity.com/paper/2012.00739/) (2020)
- [Compression Artifacts Reduction by a Deep Convolutional Network](https://mlsanity.com/paper/1504.06993/) (2015)
- [The Contextual Loss for Image Transformation with Non-Aligned Data](https://mlsanity.com/paper/1803.02077/) (2018)
- [Blind Face Restoration via Deep Multi-scale Component Dictionaries](https://mlsanity.com/paper/2008.00418/) (2020)
- [Progressive Semantic-Aware Style Transformation for Blind Face Restoration](https://mlsanity.com/paper/2009.08709/) (2020)
- [Invertible Image Rescaling](https://mlsanity.com/paper/2005.05650/) (2020)

## Built on by (most cited first, 1 of 1)

- [Perceptual Quality Improvement in Videoconferencing using Keyframes-based GAN](https://mlsanity.com/paper/2311.04263/) (2023)

## Abstract (arXiv)

Blind face restoration (BFR) is a challenging problem because of the uncertainty of the degradation patterns. This paper proposes a Restoration with Memorized Modulation (RMM) framework for universal BFR in diverse degraded scenes and heterogeneous domains. We apply random noise as well as unsupervised wavelet memory to adaptively modulate the face-enhancement generator, considering attentional denormalization in both layer and instance levels. Specifically, in the training stage, the low-level spatial feature embedding, the wavelet memory embedding obtained by wavelet transformation of the high-resolution image, as well as the disentangled high-level noise embeddings are integrated, with the guidance of attentional maps generated from layer normalization, instance normalization, and the original feature map. These three embeddings are respectively associated with the spatial content, high-frequency texture details, and a learnable universal prior against other blind image degradation patterns. We store the spatial feature of the low-resolution image and the corresponding wavelet style code as key and value in the memory unit, respectively. In the test stage, the wavelet memory value whose corresponding spatial key is the most matching with that of the inferred image is retrieved to modulate the generator. Moreover, the universal prior learned from the random noise has been memorized by training the modulation network. Experimental results show the superiority of the proposed method compared with the state-of-the-art methods, and a good generalization in the wild.