# Exploiting Deep Generative Prior for Versatile Image Restoration and Manipulation

arXiv: https://arxiv.org/abs/2003.13659 · HTML: https://mlsanity.com/paper/2003.13659/

## Key facts

- Cited by 225 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Mar 2020.

## Built on by (most cited first, 30 of 225)

- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [Denoising Diffusion Restoration Models](https://mlsanity.com/paper/2201.11793/) (2022)
- [Multi-Stage Progressive Image Restoration](https://mlsanity.com/paper/2102.02808/) (2021)
- [Zero-Shot Image Restoration Using Denoising Diffusion Null-Space Model](https://mlsanity.com/paper/2212.00490/) (2022)
- [Zero-shot Image-to-Image Translation](https://mlsanity.com/paper/2302.03027/) (2023)
- [Exploiting Diffusion Prior for Real-World Image Super-Resolution](https://mlsanity.com/paper/2305.07015/) (2023)
- [Blended Latent Diffusion](https://mlsanity.com/paper/2206.02779/) (2022)
- [Pivotal Tuning for Latent-based Editing of Real Images](https://mlsanity.com/paper/2106.05744/) (2021)
- [CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fields](https://mlsanity.com/paper/2112.05139/) (2021)
- [GAN Inversion: A Survey](https://mlsanity.com/paper/2101.05278/) (2021)
- [ResShift: Efficient Diffusion Model for Image Super-resolution by Residual Shifting](https://mlsanity.com/paper/2307.12348/) (2023)
- [Towards Real-World Blind Face Restoration with Generative Facial Prior](https://mlsanity.com/paper/2101.04061/) (2021)
- [Ablating Concepts in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2303.13516/) (2023)
- [Scaling Up to Excellence: Practicing Model Scaling for Photo-Realistic Image Restoration In the Wild](https://mlsanity.com/paper/2401.13627/) (2024)
- [DiffBIR: Towards Blind Image Restoration with Generative Diffusion Prior](https://mlsanity.com/paper/2308.15070/) (2023)
- [HyperStyle: StyleGAN Inversion with HyperNetworks for Real Image Editing](https://mlsanity.com/paper/2111.15666/) (2021)
- [Generative Diffusion Prior for Unified Image Restoration and Enhancement](https://mlsanity.com/paper/2304.01247/) (2023)
- [Pretraining is All You Need for Image-to-Image Translation](https://mlsanity.com/paper/2205.12952/) (2022)
- [Generalizing Dataset Distillation via Deep Generative Prior](https://mlsanity.com/paper/2305.01649/) (2023)
- [Diffusion Models for Image Restoration and Enhancement: A Comprehensive Survey](https://mlsanity.com/paper/2308.09388/) (2023)
- [DiffCollage: Parallel Generation of Large Content with Diffusion Models](https://mlsanity.com/paper/2303.17076/) (2023)
- [Talk-to-Edit: Fine-Grained Facial Editing via Dialog](https://mlsanity.com/paper/2109.04425/) (2021)
- [Real-World Blind Super-Resolution via Feature Matching with Implicit High-Resolution Priors](https://mlsanity.com/paper/2202.13142/) (2022)
- [AutoDIR: Automatic All-in-One Image Restoration with Latent Diffusion](https://mlsanity.com/paper/2310.10123/) (2023)
- [DifFace: Blind Face Restoration with Diffused Error Contraction](https://mlsanity.com/paper/2212.06512/) (2022)
- [Arbitrary-steps Image Super-resolution via Diffusion Inversion](https://mlsanity.com/paper/2412.09013/) (2024)
- [Constrained Graphic Layout Generation via Latent Optimization](https://mlsanity.com/paper/2108.00871/) (2021)
- [Flow-based Kernel Prior with Application to Blind Super-Resolution](https://mlsanity.com/paper/2103.15977/) (2021)
- [Auditing Privacy Defenses in Federated Learning via Generative Gradient Leakage](https://mlsanity.com/paper/2203.15696/) (2022)
- [Training-free Linear Image Inverses via Flows](https://mlsanity.com/paper/2310.04432/) (2023)

## Abstract (arXiv)

Learning a good image prior is a long-term goal for image restoration and manipulation. While existing methods like deep image prior (DIP) capture low-level image statistics, there are still gaps toward an image prior that captures rich image semantics including color, spatial coherence, textures, and high-level concepts. This work presents an effective way to exploit the image prior captured by a generative adversarial network (GAN) trained on large-scale natural images. As shown in Fig.1, the deep generative prior (DGP) provides compelling results to restore missing semantics, e.g., color, patch, resolution, of various degraded images. It also enables diverse image manipulation including random jittering, image morphing, and category transfer. Such highly flexible restoration and manipulation are made possible through relaxing the assumption of existing GAN-inversion methods, which tend to fix the generator. Notably, we allow the generator to be fine-tuned on-the-fly in a progressive manner regularized by feature distance obtained by the discriminator in GAN. We show that these easy-to-implement and practical changes help preserve the reconstruction to remain in the manifold of nature image, and thus lead to more precise and faithful reconstruction for real images. Code is available at https://github.com/XingangPan/deep-generative-prior.