# The Devil is in the Upsampling: Architectural Decisions Made Simpler for Denoising with Deep Image Prior

arXiv: https://arxiv.org/abs/2304.11409 · HTML: https://mlsanity.com/paper/2304.11409/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 40 references that are in the map.
- First posted on arXiv 22 Apr 2023.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising](https://mlsanity.com/paper/1608.03981/) (2016)
- [On the Spectral Bias of Neural Networks](https://mlsanity.com/paper/1806.08734/) (2018)
- [Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation](https://mlsanity.com/paper/2105.05537/) (2021)
- [Alias-Free Generative Adversarial Networks](https://mlsanity.com/paper/2106.12423/) (2021)
- [Deep Image Prior](https://mlsanity.com/paper/1711.10925/) (2017)
- [Leveraging Frequency Analysis for Deep Fake Image Recognition](https://mlsanity.com/paper/2003.08685/) (2020)
- [Learning Deep CNN Denoiser Prior for Image Restoration](https://mlsanity.com/paper/1704.03264/) (2017)
- [Watch your Up-Convolution: CNN Based Generative Deep Neural Networks are Failing to Reproduce Spectral Distributions](https://mlsanity.com/paper/2003.01826/) (2020)
- [Plug-and-Play ADMM for Image Restoration: Fixed Point Convergence and Applications](https://mlsanity.com/paper/1605.01710/) (2016)
- [Deep Decoder: Concise Image Representations from Untrained Non-convolutional Networks](https://mlsanity.com/paper/1810.03982/) (2018)
- ["Double-DIP": Unsupervised Image Decomposition via Coupled Deep-Image-Priors](https://mlsanity.com/paper/1812.00467/) (2018)
- [DeepRED: Deep Image Prior Powered by RED](https://mlsanity.com/paper/1903.10176/) (2019)
- [Real-world Noisy Image Denoising: A New Benchmark](https://mlsanity.com/paper/1804.02603/) (2018)
- [A Closer Look at Fourier Spectrum Discrepancies for CNN-generated Images Detection](https://mlsanity.com/paper/2103.17195/) (2021)
- [Image Restoration using Total Variation Regularized Deep Image Prior](https://mlsanity.com/paper/1810.12864/) (2018)
- [On the Frequency Bias of Generative Models](https://mlsanity.com/paper/2111.02447/) (2021)
- [A Bayesian Perspective on the Deep Image Prior](https://mlsanity.com/paper/1904.07457/) (2019)
- [Early Stopping for Deep Image Prior](https://mlsanity.com/paper/2112.06074/) (2021)
- [Training Deep Learning Based Denoisers without Ground Truth Data](https://mlsanity.com/paper/1803.01314/) (2018)
- [Denoising and Regularization via Exploiting the Structural Bias of Convolutional Generators](https://mlsanity.com/paper/1910.14634/) (2019)
- [Rethinking Deep Image Prior for Denoising](https://mlsanity.com/paper/2108.12841/) (2021)
- [On Measuring and Controlling the Spectral Bias of the Deep Image Prior](https://mlsanity.com/paper/2107.01125/) (2021)
- [The Spectral Bias of the Deep Image Prior](https://mlsanity.com/paper/1912.08905/) (2019)
- [NAS-DIP: Learning Deep Image Prior with Neural Architecture Search](https://mlsanity.com/paper/2008.11713/) (2020)
- [Neural Architecture Search for Deep Image Prior](https://mlsanity.com/paper/2001.04776/) (2020)
- [ISNAS-DIP: Image-Specific Neural Architecture Search for Deep Image Prior](https://mlsanity.com/paper/2111.15362/) (2021)

## Built on by (most cited first, 11 of 11)

- [Playing to Vision Foundation Model's Strengths in Stereo Matching](https://mlsanity.com/paper/2404.06261/) (2024)
- [GI-NAS: Boosting Gradient Inversion Attacks Through Adaptive Neural Architecture Search](https://mlsanity.com/paper/2405.20725/) (2024)
- [LighTDiff: Surgical Endoscopic Image Low-Light Enhancement with T-Diffusion](https://mlsanity.com/paper/2405.10550/) (2024)
- [YOND: Practical Blind Raw Image Denoising Free from Camera-Specific Data Dependency](https://mlsanity.com/paper/2506.03645/) (2025)
- [Towards Architecture-Agnostic Untrained Network Priors for Image Reconstruction with Frequency Regularization](https://mlsanity.com/paper/2312.09988/) (2023)
- [Masked Pre-training Enables Universal Zero-shot Denoiser](https://mlsanity.com/paper/2401.14966/) (2024)
- [Optimal Eye Surgeon: Finding Image Priors through Sparse Generators at Initialization](https://mlsanity.com/paper/2406.05288/) (2024)
- [Robustness and Exploration of Variational and Machine Learning Approaches to Inverse Problems: An Overview](https://mlsanity.com/paper/2402.12072/) (2024)
- [Zero-Shot CFC: Fast Real-World Image Denoising based on Cross-Frequency Consistency](https://mlsanity.com/paper/2510.12646/) (2025)
- [Automating Parameter Selection in Deep Image Prior for Fluorescence Microscopy Image Denoising via Similarity-Based Parameter Transfer](https://mlsanity.com/paper/2601.12055/) (2026)
- [Pooling-Based Context Modeling for Convolution-Free Deep Image Prior](https://mlsanity.com/paper/2607.02952/) (2026)

## Abstract (arXiv)

Deep Image Prior (DIP) shows that some network architectures naturally bias towards smooth images and resist noises, a phenomenon known as spectral bias. Image denoising is an immediate application of this property. Although DIP has removed the requirement of large training sets, it still presents two practical challenges for denoising: architectural design and noise-fitting, which are often intertwined. Existing methods mostly handcraft or search for the architecture from a large design space, due to the lack of understanding on how the architectural choice corresponds to the image. In this study, we analyze from a frequency perspective to demonstrate that the unlearnt upsampling is the main driving force behind the denoising phenomenon in DIP. This finding then leads to strategies for estimating a suitable architecture for every image without a laborious search. Extensive experiments show that the estimated architectures denoise and preserve the textural details better than current methods with up to 95% fewer parameters. The under-parameterized nature also makes them especially robust to a higher level of noise.