# Scaling Laws For Deep Learning Based Image Reconstruction

arXiv: https://arxiv.org/abs/2209.13435 · HTML: https://mlsanity.com/paper/2209.13435/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 73 references that are in the map.
- First posted on arXiv 27 Sep 2022.

## Builds on (most cited first, 30 of 38)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network](https://mlsanity.com/paper/1609.04802/) (2016)
- [SwinIR: Image Restoration Using Swin Transformer](https://mlsanity.com/paper/2108.10257/) (2021)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising](https://mlsanity.com/paper/1608.03981/) (2016)
- [Image Super-Resolution Using Deep Convolutional Networks](https://mlsanity.com/paper/1501.00092/) (2014)
- [Instance Normalization: The Missing Ingredient for Fast Stylization](https://mlsanity.com/paper/1607.08022/) (2016)
- [Scaling Vision Transformers](https://mlsanity.com/paper/2106.04560/) (2021)
- [Pre-Trained Image Processing Transformer](https://mlsanity.com/paper/2012.00364/) (2020)
- [Deep Learning Scaling is Predictable, Empirically](https://mlsanity.com/paper/1712.00409/) (2017)
- [fastMRI: An Open Dataset and Benchmarks for Accelerated MRI](https://mlsanity.com/paper/1811.08839/) (2018)
- [Plug-and-Play Image Restoration with Deep Denoiser Prior](https://mlsanity.com/paper/2008.13751/) (2020)
- [Deep Convolutional Neural Network for Inverse Problems in Imaging](https://mlsanity.com/paper/1611.03679/) (2016)
- [An Introduction to Matrix Concentration Inequalities](https://mlsanity.com/paper/1501.01571/) (2015)
- [Explaining Neural Scaling Laws](https://mlsanity.com/paper/2102.06701/) (2021)
- [Learning a Variational Network for Reconstruction of Accelerated MRI Data](https://mlsanity.com/paper/1704.00447/) (2017)
- [Unprocessing Images for Learned Raw Denoising](https://mlsanity.com/paper/1811.11127/) (2018)
- [Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalization](https://mlsanity.com/paper/2107.04649/) (2021)
- [A Constructive Prediction of the Generalization Error Across Scales](https://mlsanity.com/paper/1909.12673/) (2019)
- [End-to-End Variational Networks for Accelerated MRI Reconstruction](https://mlsanity.com/paper/2004.06688/) (2020)
- [High-Quality Self-Supervised Deep Image Denoising](https://mlsanity.com/paper/1901.10277/) (2019)
- [Scaling Laws for Neural Machine Translation](https://mlsanity.com/paper/2109.07740/) (2021)
- [Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIP](https://mlsanity.com/paper/2208.05516/) (2022)
- [Robust and interpretable blind image denoising via bias-free convolutional neural networks](https://mlsanity.com/paper/1906.05478/) (2019)
- [Practical Blind Image Denoising via Swin-Conv-UNet and Data Synthesis](https://mlsanity.com/paper/2203.13278/) (2022)

## Built on by (most cited first, 14 of 14)

- [eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers](https://mlsanity.com/paper/2211.01324/) (2022)
- [Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit](https://mlsanity.com/paper/2309.16620/) (2023)
- [Uncovering Neural Scaling Laws in Molecular Representation Learning](https://mlsanity.com/paper/2309.15123/) (2023)
- [Turbulence in Focus: Benchmarking Scaling Behavior of 3D Volumetric Super-Resolution with BLASTNet 2.0 Data](https://mlsanity.com/paper/2309.13457/) (2023)
- [Model Merging Scaling Laws in Large Language Models](https://mlsanity.com/paper/2509.24244/) (2025)
- [Analysis of Deep Image Prior and Exploiting Self-Guidance for Image Reconstruction](https://mlsanity.com/paper/2402.04097/) (2024)
- [Robustness of Deep Learning for Accelerated MRI: Benefits of Diverse Training Data](https://mlsanity.com/paper/2312.10271/) (2023)
- [Analyzing the Sample Complexity of Self-Supervised Image Reconstruction Methods](https://mlsanity.com/paper/2305.19079/) (2023)
- [Scaling Learned Image Compression Models up to 1 Billion](https://mlsanity.com/paper/2508.09075/) (2025)
- [Finding the Reflection Point: Unpadding Images to Remove Data Augmentation Artifacts in Large Open Source Image Datasets for Machine Learning](https://mlsanity.com/paper/2504.03168/) (2025)
- [Out-of-distribution data supervision towards biomedical semantic segmentation](https://mlsanity.com/paper/2507.12105/) (2025)
- [Instance-Wise Adaptive Sampling for Dataset Construction in Approximating Inverse Problem Solutions](https://mlsanity.com/paper/2509.04583/) (2025)
- [Unify Variables in Neural Scaling Laws for General Audio Representations via Embedding Effective Rank](https://mlsanity.com/paper/2510.10948/) (2025)
- [Compute-Optimal Network Design for Echocardiography Myocardial Segmentation and Perfusion Quantification using Neural Scaling Laws](https://mlsanity.com/paper/2606.06725/) (2026)

## Abstract (arXiv)

Deep neural networks trained end-to-end to map a measurement of a (noisy) image to a clean image perform excellent for a variety of linear inverse problems. Current methods are only trained on a few hundreds or thousands of images as opposed to the millions of examples deep networks are trained on in other domains. In this work, we study whether major performance gains are expected from scaling up the training set size. We consider image denoising, accelerated magnetic resonance imaging, and super-resolution and empirically determine the reconstruction quality as a function of training set size, while simultaneously scaling the network size. For all three tasks we find that an initially steep power-law scaling slows significantly already at moderate training set sizes. Interpolating those scaling laws suggests that even training on millions of images would not significantly improve performance. To understand the expected behavior, we analytically characterize the performance of a linear estimator learned with early stopped gradient descent. The result formalizes the intuition that once the error induced by learning the signal model is small relative to the error floor, more training examples do not improve performance.