# Deep Learning meets Nonparametric Regression: Are Weight-Decayed DNNs Locally Adaptive? arXiv: https://arxiv.org/abs/2204.09664 · HTML: https://mlsanity.com/paper/2204.09664/ ## Key facts - Cited by 10 papers in the map (data through 14 Oct 2026). - Builds on 31 of its 69 references that are in the map. - First posted on arXiv 20 Apr 2022. ## Builds on (most cited first, 30 of 31) - [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016) - [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018) - [Deep Double Descent: Where Bigger Models and More Data Hurt](https://mlsanity.com/paper/1912.02292/) (2019) - [Error bounds for approximations with deep ReLU networks](https://mlsanity.com/paper/1610.01145/) (2016) - [On Exact Computation with an Infinitely Wide Neural Net](https://mlsanity.com/paper/1904.11955/) (2019) - [Nonparametric regression using deep neural networks with ReLU activation function](https://mlsanity.com/paper/1708.06633/) (2017) - [In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learning](https://mlsanity.com/paper/1412.6614/) (2014) - [The Loss Surfaces of Multilayer Networks](https://mlsanity.com/paper/1412.0233/) (2014) - [The large learning rate phase of deep learning: the catapult mechanism](https://mlsanity.com/paper/2003.02218/) (2020) - [To understand deep learning we need to understand kernel learning](https://mlsanity.com/paper/1802.01396/) (2018) - [Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionality](https://mlsanity.com/paper/1810.08033/) (2018) - [When Do Neural Networks Outperform Kernel Methods?](https://mlsanity.com/paper/2006.13409/) (2020) - [A Function Space View of Bounded Norm Infinite Width ReLU Nets: The Multivariate Case](https://mlsanity.com/paper/1910.01635/) (2019) - [How do infinite width bounded norm networks look in function space?](https://mlsanity.com/paper/1902.05040/) (2019) - [Neural Networks are Convex Regularizers: Exact Polynomial-time Convex Optimization Formulations for Two-layer Networks](https://mlsanity.com/paper/2002.10553/) (2020) - [Dropout Training as Adaptive Regularization](https://mlsanity.com/paper/1307.1493/) (2013) - [What Can ResNet Learn Efficiently, Going Beyond Kernels?](https://mlsanity.com/paper/1905.10337/) (2019) - [What Kinds of Functions do Deep Neural Networks Learn? Insights from Variational Spline Theory](https://mlsanity.com/paper/2105.03361/) (2021) - [Revealing the Structure of Deep Neural Networks via Convex Duality](https://mlsanity.com/paper/2002.09773/) (2020) - [Convex Geometry and Duality of Over-parameterized Neural Networks](https://mlsanity.com/paper/2002.11219/) (2020) - [Adaptive piecewise polynomial estimation via trend filtering](https://mlsanity.com/paper/1304.2986/) (2013) - [Near-Minimax Optimal Estimation With Shallow ReLU Neural Networks](https://mlsanity.com/paper/2109.08844/) (2021) - [Approximation and Non-parametric Estimation of ResNet-type Convolutional Neural Networks](https://mlsanity.com/paper/1903.10047/) (2019) - [Besov Function Approximation and Binary Classification on Low-Dimensional Manifolds Using Convolutional Residual Networks](https://mlsanity.com/paper/2109.02832/) (2021) - [Online Forecasting of Total-Variation-bounded Sequences](https://mlsanity.com/paper/1906.03364/) (2019) - [Statistical Guarantees for Regularized Neural Networks](https://mlsanity.com/paper/2006.00294/) (2020) - [Global Optimality Beyond Two Layers: Training Deep ReLU Networks via Convex Programs](https://mlsanity.com/paper/2110.05518/) (2021) - [Path Regularization: A Convexity and Sparsity Inducing Regularization for Parallel ReLU Networks](https://mlsanity.com/paper/2110.09548/) (2021) - [Adaptive Online Estimation of Piecewise Polynomial Trends](https://mlsanity.com/paper/2010.00073/) (2020) - [The Falling Factorial Basis and Its Statistical Applications](https://mlsanity.com/paper/1405.0558/) (2014) ## Built on by (most cited first, 10 of 10) - [The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformers](https://mlsanity.com/paper/2210.06313/) (2022) - [Nonparametric regression using over-parameterized shallow ReLU neural networks](https://mlsanity.com/paper/2306.08321/) (2023) - [Stable Minima Cannot Overfit in Univariate ReLU Networks: Generalization by Large Step Sizes](https://mlsanity.com/paper/2406.06838/) (2024) - [Intrinsic dimensionality and generalization properties of the R-norm inductive bias](https://mlsanity.com/paper/2206.05317/) (2022) - [How many samples are needed to train a deep neural network?](https://mlsanity.com/paper/2405.16696/) (2024) - [Generalization Below the Edge of Stability: The Role of Data Geometry](https://mlsanity.com/paper/2510.18120/) (2025) - [Nonparametric Classification on Low Dimensional Manifolds using Overparameterized Convolutional Residual Networks](https://mlsanity.com/paper/2307.01649/) (2023) - [How Inverse Conditional Flows Can Serve as a Substitute for Distributional Regression](https://mlsanity.com/paper/2405.05429/) (2024) - [Gradient Descent Finds Over-Parameterized Neural Networks with Sharp Generalization for Nonparametric Regression](https://mlsanity.com/paper/2411.02904/) (2024) - [Characterizing 4-string contact interaction using machine learning](https://mlsanity.com/paper/2211.09129/) (2022) ## Abstract (arXiv) We study the theory of neural network (NN) from the lens of classical nonparametric regression problems with a focus on NN's ability to adaptively estimate functions with heterogeneous smoothness -- a property of functions in Besov or Bounded Variation (BV) classes. Existing work on this problem requires tuning the NN architecture based on the function spaces and sample size. We consider a "Parallel NN" variant of deep ReLU networks and show that the standard ℓ2 regularization is equivalent to promoting the ℓp-sparsity (0