# On Measuring and Controlling the Spectral Bias of the Deep Image Prior

arXiv: https://arxiv.org/abs/2107.01125 · HTML: https://mlsanity.com/paper/2107.01125/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 78 references that are in the map.
- First posted on arXiv 2 Jul 2021.

## Builds on (most cited first, 30 of 30)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network](https://mlsanity.com/paper/1609.04802/) (2016)
- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising](https://mlsanity.com/paper/1608.03981/) (2016)
- [On the Spectral Bias of Neural Networks](https://mlsanity.com/paper/1806.08734/) (2018)
- [Image Super-Resolution Using Deep Convolutional Networks](https://mlsanity.com/paper/1501.00092/) (2014)
- [Deep Image Prior](https://mlsanity.com/paper/1711.10925/) (2017)
- [FFDNet: Toward a Fast and Flexible Solution for CNN based Image Denoising](https://mlsanity.com/paper/1710.04026/) (2017)
- [Deep Convolutional Neural Network for Inverse Problems in Imaging](https://mlsanity.com/paper/1611.03679/) (2016)
- [Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks](https://mlsanity.com/paper/1901.06523/) (2019)
- [Trainable Nonlinear Reaction Diffusion: A Flexible Framework for Fast and Effective Image Restoration](https://mlsanity.com/paper/1508.02848/) (2015)
- [Fast and Accurate Image Super-Resolution with Deep Laplacian Pyramid Networks](https://mlsanity.com/paper/1710.01992/) (2017)
- [Compression Artifacts Reduction by a Deep Convolutional Network](https://mlsanity.com/paper/1504.06993/) (2015)
- [Deep Decoder: Concise Image Representations from Untrained Non-convolutional Networks](https://mlsanity.com/paper/1810.03982/) (2018)
- ["Double-DIP": Unsupervised Image Decomposition via Coupled Deep-Image-Priors](https://mlsanity.com/paper/1812.00467/) (2018)
- [Bringing Old Photos Back to Life](https://mlsanity.com/paper/2004.09484/) (2020)
- [DeepRED: Deep Image Prior Powered by RED](https://mlsanity.com/paper/1903.10176/) (2019)
- [A Review of Convolutional Neural Networks for Inverse Problems in Imaging](https://mlsanity.com/paper/1710.04011/) (2017)
- [Image Restoration using Total Variation Regularized Deep Image Prior](https://mlsanity.com/paper/1810.12864/) (2018)
- [A Bayesian Perspective on the Deep Image Prior](https://mlsanity.com/paper/1904.07457/) (2019)
- [Denoising and Regularization via Exploiting the Structural Bias of Convolutional Generators](https://mlsanity.com/paper/1910.14634/) (2019)
- [The Spectral Bias of the Deep Image Prior](https://mlsanity.com/paper/1912.08905/) (2019)
- [NAS-DIP: Learning Deep Image Prior with Neural Architecture Search](https://mlsanity.com/paper/2008.11713/) (2020)
- [Neural Architecture Search for Deep Image Prior](https://mlsanity.com/paper/2001.04776/) (2020)
- [A Frequency Domain Neural Network for Fast Image Super-resolution](https://mlsanity.com/paper/1712.03037/) (2017)
- [Unsharp Mask Guided Filtering](https://mlsanity.com/paper/2106.01428/) (2021)
- [BP-DIP: A Backprojection based Deep Image Prior](https://mlsanity.com/paper/2003.05417/) (2020)

## Built on by (most cited first, 30 of 36)

- [Paint-it: Text-to-Texture Synthesis via Deep Convolutional Texture Map Optimization and Physically-Based Rendering](https://mlsanity.com/paper/2312.11360/) (2023)
- [DMPlug: A Plug-in Method for Solving Inverse Problems with Diffusion Models](https://mlsanity.com/paper/2405.16749/) (2024)
- [Early Stopping for Deep Image Prior](https://mlsanity.com/paper/2112.06074/) (2021)
- [Maximum Class Separation as Inductive Bias in One Matrix](https://mlsanity.com/paper/2206.08704/) (2022)
- [Self-Validation: Early Stopping for Single-Instance Deep Generative Priors](https://mlsanity.com/paper/2110.12271/) (2021)
- [ISNAS-DIP: Image-Specific Neural Architecture Search for Deep Image Prior](https://mlsanity.com/paper/2111.15362/) (2021)
- [The Devil is in the Upsampling: Architectural Decisions Made Simpler for Denoising with Deep Image Prior](https://mlsanity.com/paper/2304.11409/) (2023)
- [Analysis of Deep Image Prior and Exploiting Self-Guidance for Image Reconstruction](https://mlsanity.com/paper/2402.04097/) (2024)
- [Fast Training of Sinusoidal Neural Fields via Scaling Initialization](https://mlsanity.com/paper/2410.04779/) (2024)
- [Uncertainty Estimation for Computed Tomography with a Linearised Deep Image Prior](https://mlsanity.com/paper/2203.00479/) (2022)
- [Blind Image Deblurring with Unknown Kernel Size and Substantial Noise](https://mlsanity.com/paper/2208.09483/) (2022)
- [Turb-L1: Achieving Long-term Turbulence Tracing By Tackling Spectral Bias](https://mlsanity.com/paper/2505.19038/) (2025)
- [A Baseline Method for Removing Invisible Image Watermarks using Deep Image Prior](https://mlsanity.com/paper/2502.13998/) (2025)
- [Recurrent Spike-based Image Restoration under General Illumination](https://mlsanity.com/paper/2308.03018/) (2023)
- [Highly Accelerated MRI via Implicit Neural Representation Guided Posterior Sampling of Diffusion Models](https://mlsanity.com/paper/2407.02744/) (2024)
- [PIP: Positional-encoding Image Prior](https://mlsanity.com/paper/2211.14298/) (2022)
- [SwinGar: Spectrum-Inspired Neural Dynamic Deformation for Free-Swinging Garments](https://mlsanity.com/paper/2308.02827/) (2023)
- [Convergence and Recovery Guarantees of Unsupervised Neural Networks for Inverse Problems](https://mlsanity.com/paper/2309.12128/) (2023)
- [Towards Architecture-Agnostic Untrained Network Priors for Image Reconstruction with Frequency Regularization](https://mlsanity.com/paper/2312.09988/) (2023)
- [Masked Pre-training Enables Universal Zero-shot Denoiser](https://mlsanity.com/paper/2401.14966/) (2024)
- [Convergence Guarantees of Overparametrized Wide Deep Inverse Prior](https://mlsanity.com/paper/2303.11265/) (2023)
- [SVD-DIP: Overcoming the Overfitting Problem in DIP-based CT Reconstruction](https://mlsanity.com/paper/2303.15748/) (2023)
- [Optimal Eye Surgeon: Finding Image Priors through Sparse Generators at Initialization](https://mlsanity.com/paper/2406.05288/) (2024)
- [Fourier Sensitivity and Regularization of Computer Vision Models](https://mlsanity.com/paper/2301.13514/) (2023)
- [Image Reconstruction via Deep Image Prior Subspaces](https://mlsanity.com/paper/2302.10279/) (2023)
- [MCDIP-ADMM: Overcoming Overfitting in DIP-based CT reconstruction](https://mlsanity.com/paper/2304.03895/) (2023)
- [Low-light phase retrieval with implicit generative priors](https://mlsanity.com/paper/2402.17745/) (2024)
- [Physics-Driven Neural Compensation For Electrical Impedance Tomography](https://mlsanity.com/paper/2504.18067/) (2025)
- [Novel View Synthesis of Humans using Differentiable Rendering](https://mlsanity.com/paper/2303.15880/) (2023)
- [Neural Gradient Regularizer](https://mlsanity.com/paper/2308.16612/) (2023)

## Abstract (arXiv)

The deep image prior showed that a randomly initialized network with a suitable architecture can be trained to solve inverse imaging problems by simply optimizing it's parameters to reconstruct a single degraded image. However, it suffers from two practical limitations. First, it remains unclear how to control the prior beyond the choice of the network architecture. Second, training requires an oracle stopping criterion as during the optimization the performance degrades after reaching an optimum value. To address these challenges we introduce a frequency-band correspondence measure to characterize the spectral bias of the deep image prior, where low-frequency image signals are learned faster and better than high-frequency counterparts. Based on our observations, we propose techniques to prevent the eventual performance degradation and accelerate convergence. We introduce a Lipschitz-controlled convolution layer and a Gaussian-controlled upsampling layer as plug-in replacements for layers used in the deep architectures. The experiments show that with these changes the performance does not degrade during optimization, relieving us from the need for an oracle stopping criterion. We further outline a stopping criterion to avoid superfluous computation. Finally, we show that our approach obtains favorable results compared to current approaches across various denoising, deblocking, inpainting, super-resolution and detail enhancement tasks. Code is available at urlhttps://github.com/shizenglin/Measure-and-Control-Spectral-Bias.