mlsanityData to 14 Oct 2026

How do Minimum-Norm Shallow Denoisers Look in Function Space?

arXiv 2311.06748 ↗First posted 12 Nov 2023stat.ML cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

Neural network (NN) denoisers are an essential building block in many common tasks, ranging from image reconstruction to image generation. However, the success of these models is not well understood from a theoretical perspective. In this paper, we aim to characterize the functions realized by shallow ReLU NN denoisers -- in the common theoretical setting of interpolation (i.e., zero training loss) with a minimal representation cost (i.e., minimal ℓ2 norm weights). First, for univariate data, we derive a closed form for the NN denoiser function, find it is contractive toward the clean data points, and prove it generalizes better than the empirical MMSE estimator at a low noise level. Next, for multivariate data, we find the NN denoiser functions in a closed form under various geometric assumptions on the training data: data contained in a low-dimensional subspace, data contained in a union of one-sided rays, or several types of simplexes. These functions decompose into a sum of simple rank-one piecewise linear interpolations aligned with edges and/or faces connecting training samples. We empirically verify this alignment phenomenon on synthetic data and real images.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 18 of its 31 references that are in the map. First posted on arXiv 12 Nov 2023.

Builds on

18 of 31 references.

References · 18

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  3. 2016Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoisingcited by 1,358
  4. 2017Exploring Generalization in Deep Learningcited by 600
  5. 2020Plug-and-Play Image Restoration with Deep Denoiser Priorcited by 470
  6. 2015Norm-Based Capacity Control in Neural Networkscited by 299
  7. 2019A Function Space View of Bounded Norm Infinite Width ReLU Nets: The Multivariate Casecited by 114
  8. 2019How do infinite width bounded norm networks look in function space?cited by 106
  9. 2023Spontaneous Symmetry Breaking in Generative Diffusion Modelscited by 80
  10. 2023Image Denoising: The Deep Learning Revolution and Beyond -- A Survey Paper --cited by 71
  11. 2021What Kinds of Functions do Deep Neural Networks Learn? Insights from Variational Spline Theorycited by 60
  12. 2020Convex Geometry and Duality of Over-parameterized Neural Networkscited by 49
6 more
  1. 2022Implicit Bias of Large Depth Networks: a Notion of Rank for Nonlinear Functionscited by 38
  2. 2023Penalising the biases in norm regularisation enforces sparsitycited by 19
  3. 2022Feature Learning in L2-regularized DNNs: Attraction/Repulsion and Sparsitycited by 17
  4. 2020Convex Regularization Behind Neural Reconstructioncited by 15
  5. 2023The Implicit Bias of Minima Stability in Multivariate Shallow ReLU Networkscited by 15
  6. 2021Ridgeless Interpolation with Shallow ReLU Networks in 1D is Nearest Neighbor Curvature Extrapolation and Provably Generalizes on Lipschitz Functionscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 3; 2026: 1.

9 papers.

  1. 2024Understanding Generalizability of Diffusion Models Requires Rethinking the Hidden Gaussian Structurecited by 38
  2. 2024How Uniform Random Weights Induce Non-uniform Bias: Typical Interpolating Neural Networks Generalize with Narrow Teacherscited by 18
  3. 2025Taking a Big Step: Large Learning Rates in Denoising Score Matching Prevent Memorizationcited by 17
  4. 2025Generalization of Diffusion Models Arises with a Balanced Representation Spacecited by 11
  5. 2024Analyzing Neural Network-Based Generative Diffusion Models through Convex Optimizationcited by 7
  6. 2025When Diffusion Models Memorize: Inductive Biases in Probability Flow of Minimum-Norm Shallow Neural Netscited by 6
  7. 2024ReLUs Are Sufficient for Learning Implicit Neural Representationscited by 5
  8. 2026Two Calm Ends and the Wild Middle: A Geometric Picture of Memorization in Diffusion Modelscited by 3
  9. 2024The Effects of Multi-Task Learning on ReLU Neural Network Functionscited by 0