mlsanityData to 14 Oct 2026

Deep Learning meets Nonparametric Regression: Are Weight-Decayed DNNs Locally Adaptive?

arXiv 2204.09664 ↗First posted 20 Apr 2022cs.LG stat.MLcited by 10

Not on the map. Why?

Abstract (arXiv)

We study the theory of neural network (NN) from the lens of classical nonparametric regression problems with a focus on NN's ability to adaptively estimate functions with heterogeneous smoothness -- a property of functions in Besov or Bounded Variation (BV) classes. Existing work on this problem requires tuning the NN architecture based on the function spaces and sample size. We consider a "Parallel NN" variant of deep ReLU networks and show that the standard ℓ2 regularization is equivalent to promoting the ℓp-sparsity (0<p<1) in the coefficient vector of an end-to-end learned function bases, i.e., a dictionary. Using this equivalence, we further establish that by tuning only the regularization factor, such parallel NN achieves an estimation error arbitrarily close to the minimax rates for both the Besov and BV classes. Notably, it gets exponentially closer to minimax optimal as the NN gets deeper. Our research sheds new lights on why depth matters and how NNs are more powerful than kernel methods.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 31 of its 69 references that are in the map. First posted on arXiv 20 Apr 2022.

Builds on

31 of 69 references.

References · 31

  1. 2016Wide Residual Networkscited by 3,070
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  3. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 672
  4. 2016Error bounds for approximations with deep ReLU networkscited by 574
  5. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
  6. 2017Nonparametric regression using deep neural networks with ReLU activation functioncited by 433
  7. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 342
  8. 2014The Loss Surfaces of Multilayer Networkscited by 280
  9. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 216
  10. 2018To understand deep learning we need to understand kernel learningcited by 187
  11. 2018Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionalitycited by 162
  12. 2020When Do Neural Networks Outperform Kernel Methods?cited by 150
19 more
  1. 2019A Function Space View of Bounded Norm Infinite Width ReLU Nets: The Multivariate Casecited by 114
  2. 2019How do infinite width bounded norm networks look in function space?cited by 106
  3. 2020Neural Networks are Convex Regularizers: Exact Polynomial-time Convex Optimization Formulations for Two-layer Networkscited by 98
  4. 2013Dropout Training as Adaptive Regularizationcited by 93
  5. 2019What Can ResNet Learn Efficiently, Going Beyond Kernels?cited by 91
  6. 2021What Kinds of Functions do Deep Neural Networks Learn? Insights from Variational Spline Theorycited by 60
  7. 2020Revealing the Structure of Deep Neural Networks via Convex Dualitycited by 54
  8. 2020Convex Geometry and Duality of Over-parameterized Neural Networkscited by 49
  9. 2013Adaptive piecewise polynomial estimation via trend filteringcited by 47
  10. 2021Near-Minimax Optimal Estimation With Shallow ReLU Neural Networkscited by 41
  11. 2019Approximation and Non-parametric Estimation of ResNet-type Convolutional Neural Networkscited by 40
  12. 2021Besov Function Approximation and Binary Classification on Low-Dimensional Manifolds Using Convolutional Residual Networkscited by 32
  13. 2019Online Forecasting of Total-Variation-bounded Sequencescited by 31
  14. 2020Statistical Guarantees for Regularized Neural Networkscited by 25
  15. 2021Global Optimality Beyond Two Layers: Training Deep ReLU Networks via Convex Programscited by 25
  16. 2021Path Regularization: A Convexity and Sparsity Inducing Regularization for Parallel ReLU Networkscited by 14
  17. 2020Adaptive Online Estimation of Piecewise Polynomial Trendscited by 11
  18. 2014The Falling Factorial Basis and Its Statistical Applicationscited by 8
  19. 2021Multivariate Trend Filtering for Lattice Datacited by 4

Built on by

Cited by year

Citing papers per year: 2022: 3; 2023: 2; 2024: 4; 2025: 1.

10 papers.

  1. 2022The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformerscited by 122
  2. 2023Nonparametric regression using over-parameterized shallow ReLU neural networkscited by 24
  3. 2024Stable Minima Cannot Overfit in Univariate ReLU Networks: Generalization by Large Step Sizescited by 12
  4. 2022Intrinsic dimensionality and generalization properties of the R-norm inductive biascited by 5
  5. 2024How many samples are needed to train a deep neural network?cited by 5
  6. 2025Generalization Below the Edge of Stability: The Role of Data Geometrycited by 3
  7. 2023Nonparametric Classification on Low Dimensional Manifolds using Overparameterized Convolutional Residual Networkscited by 2
  8. 2024How Inverse Conditional Flows Can Serve as a Substitute for Distributional Regressioncited by 2
  9. 2024Gradient Descent Finds Over-Parameterized Neural Networks with Sharp Generalization for Nonparametric Regressioncited by 2
  10. 2022Characterizing 4-string contact interaction using machine learningcited by 0