# Intermediate Layer Optimization for Inverse Problems using Deep Generative Models

arXiv: https://arxiv.org/abs/2102.07364 · HTML: https://mlsanity.com/paper/2102.07364/

## Key facts

- Cited by 70 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 74 references that are in the map.
- First posted on arXiv 15 Feb 2021.

## Builds on (most cited first, 30 of 40)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation](https://mlsanity.com/paper/2008.00951/) (2020)
- [Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?](https://mlsanity.com/paper/1904.03189/) (2019)
- [MaskGAN: Towards Diverse and Interactive Facial Image Manipulation](https://mlsanity.com/paper/1907.11922/) (2019)
- [Free-Form Image Inpainting with Gated Convolution](https://mlsanity.com/paper/1806.03589/) (2018)
- [Generative Image Inpainting with Contextual Attention](https://mlsanity.com/paper/1801.07892/) (2018)
- [PULSE: Self-Supervised Photo Upsampling via Latent Space Exploration of Generative Models](https://mlsanity.com/paper/2003.03808/) (2020)
- [Compressed Sensing using Generative Models](https://mlsanity.com/paper/1703.03208/) (2017)
- [Deep Learning Techniques for Inverse Problems in Imaging](https://mlsanity.com/paper/2005.06001/) (2020)
- [Generating Adversarial Examples with Adversarial Networks](https://mlsanity.com/paper/1801.02610/) (2018)
- [DeepFakes: a New Threat to Face Recognition? Assessment and Detection](https://mlsanity.com/paper/1812.08685/) (2018)
- [Deep Learning for Single Image Super-Resolution: A Brief Review](https://mlsanity.com/paper/1808.03344/) (2018)
- [Seeing What a GAN Cannot Generate](https://mlsanity.com/paper/1910.11626/) (2019)
- [Deep Learning on Image Denoising: An overview](https://mlsanity.com/paper/1912.13171/) (2019)
- [Deep Learning for Deepfakes Creation and Detection: A Survey](https://mlsanity.com/paper/1909.11573/) (2019)
- [Coherent Semantic Attention for Image Inpainting](https://mlsanity.com/paper/1905.12384/) (2019)
- [Invertible generative models for inverse problems: mitigating representation error and dataset bias](https://mlsanity.com/paper/1905.11672/) (2019)
- [Phase Retrieval Under a Generative Prior](https://mlsanity.com/paper/1807.04261/) (2018)
- [One Network to Solve Them All --- Solving Linear Inverse Problems using Deep Projection Models](https://mlsanity.com/paper/1703.09912/) (2017)
- [Solving Linear Inverse Problems Using GAN Priors: An Algorithm with Provable Guarantees](https://mlsanity.com/paper/1802.08406/) (2018)
- [Learned Image Downscaling for Upscaling using Content Adaptive Resampler](https://mlsanity.com/paper/1907.12904/) (2019)
- [Deep Compressed Sensing](https://mlsanity.com/paper/1905.06723/) (2019)
- [Improving the Fairness of Deep Generative Models without Retraining](https://mlsanity.com/paper/2012.04842/) (2020)
- [Global Guarantees for Enforcing Deep Generative Priors by Empirical Risk](https://mlsanity.com/paper/1705.07576/) (2017)
- [Modeling Sparse Deviations for Compressed Sensing using Generative Models](https://mlsanity.com/paper/1807.01442/) (2018)

## Built on by (most cited first, 30 of 70)

- [Denoising Diffusion Restoration Models](https://mlsanity.com/paper/2201.11793/) (2022)
- [Improving Diffusion Models for Inverse Problems using Manifold Constraints](https://mlsanity.com/paper/2206.00941/) (2022)
- [Robust Compressed Sensing MRI with Deep Generative Priors](https://mlsanity.com/paper/2108.01368/) (2021)
- [DiffBIR: Towards Blind Image Restoration with Generative Diffusion Prior](https://mlsanity.com/paper/2308.15070/) (2023)
- [A Survey on Diffusion Models for Inverse Problems](https://mlsanity.com/paper/2410.00083/) (2024)
- [D2C: Diffusion-Denoising Models for Few-shot Conditional Generation](https://mlsanity.com/paper/2106.06819/) (2021)
- [Training-free Linear Image Inverses via Flows](https://mlsanity.com/paper/2310.04432/) (2023)
- [FuseDream: Training-Free Text-to-Image Generation with Improved CLIP+GAN Space Optimization](https://mlsanity.com/paper/2112.01573/) (2021)
- [Generative Joint Source-Channel Coding for Semantic Image Transmission](https://mlsanity.com/paper/2211.13772/) (2022)
- [Labels4Free: Unsupervised Segmentation using StyleGAN](https://mlsanity.com/paper/2103.14968/) (2021)
- [MIMO Channel Estimation using Score-Based Generative Models](https://mlsanity.com/paper/2204.07122/) (2022)
- [GIFD: A Generative Gradient Inversion Method with Feature Domain Optimization](https://mlsanity.com/paper/2308.04699/) (2023)
- [Learning to Solve PDE-constrained Inverse Problems with Graph Networks](https://mlsanity.com/paper/2206.00711/) (2022)
- [Multiresolution Textual Inversion](https://mlsanity.com/paper/2211.17115/) (2022)
- [Holo-Relighting: Controllable Volumetric Portrait Relighting from a Single Image](https://mlsanity.com/paper/2403.09632/) (2024)
- [Generative Adversarial Registration for Improved Conditional Deformable Templates](https://mlsanity.com/paper/2105.04349/) (2021)
- [Blind Image Restoration via Fast Diffusion Inversion](https://mlsanity.com/paper/2405.19572/) (2024)
- [Idempotence and Perceptual Image Compression](https://mlsanity.com/paper/2401.08920/) (2024)
- [Regularising Inverse Problems with Generative Machine Learning Models](https://mlsanity.com/paper/2107.11191/) (2021)
- [Theoretical Perspectives on Deep Learning Methods in Inverse Problems](https://mlsanity.com/paper/2206.14373/) (2022)
- [Prior Image-Constrained Reconstruction using Style-Based Generative Models](https://mlsanity.com/paper/2102.12525/) (2021)
- [A Closer Look at GAN Priors: Exploiting Intermediate Features for Enhanced Model Inversion Attacks](https://mlsanity.com/paper/2407.13863/) (2024)
- [LATINO-PRO: LAtent consisTency INverse sOlver with PRompt Optimization](https://mlsanity.com/paper/2503.12615/) (2025)
- [Designing Counterfactual Generators using Deep Model Inversion](https://mlsanity.com/paper/2109.14274/) (2021)
- [Repulsive Latent Score Distillation for Solving Inverse Problems](https://mlsanity.com/paper/2406.16683/) (2024)
- [Robust Unsupervised StyleGAN Image Restoration](https://mlsanity.com/paper/2302.06733/) (2023)
- [CommIN: Semantic Image Communications as an Inverse Problem with INN-Guided Diffusion Models](https://mlsanity.com/paper/2310.01130/) (2023)
- [EdiBERT, a generative model for image editing](https://mlsanity.com/paper/2111.15264/) (2021)
- [TurbuGAN: An Adversarial Learning Approach to Spatially-Varying Multiframe Blind Deconvolution with Applications to Imaging Through Turbulence](https://mlsanity.com/paper/2203.06764/) (2022)
- [Score-Guided Intermediate Layer Optimization: Fast Langevin Mixing for Inverse Problems](https://mlsanity.com/paper/2206.09104/) (2022)

## Abstract (arXiv)

We propose Intermediate Layer Optimization (ILO), a novel optimization algorithm for solving inverse problems with deep generative models. Instead of optimizing only over the initial latent code, we progressively change the input layer obtaining successively more expressive generators. To explore the higher dimensional spaces, our method searches for latent codes that lie within a small l1 ball around the manifold induced by the previous layer. Our theoretical analysis shows that by keeping the radius of the ball relatively small, we can improve the established error bound for compressed sensing with deep generative models. We empirically show that our approach outperforms state-of-the-art methods introduced in StyleGAN-2 and PULSE for a wide range of inverse problems including inpainting, denoising, super-resolution and compressed sensing.