mlsanityData to 14 Oct 2026

Intrinsic dimensionality and generalization properties of the R-norm inductive bias

arXiv 2206.05317 ↗First posted 10 Jun 2022cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

We study the structural and statistical properties of R-norm minimizing interpolants of datasets labeled by specific target functions. The R-norm is the basis of an inductive bias for two-layer neural networks, recently introduced to capture the functional effect of controlling the size of network weights, independently of the network width. We find that these interpolants are intrinsically multivariate functions, even when there are ridge functions that fit the data, and also that the R-norm inductive bias is not sufficient for achieving statistically optimal generalization for certain learning problems. Altogether, these results shed new light on an inductive bias that is connected to practical neural network training.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 33 of its 76 references that are in the map. First posted on arXiv 10 Jun 2022.

Builds on

33 of 76 references.

References · 33

  1. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  2. 2017Exploring Generalization in Deep Learningcited by 600
  3. 2017Nonparametric regression using deep neural networks with ReLU activation functioncited by 433
  4. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 342
  5. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 331
  6. 2016Benefits of depth in neural networkscited by 250
  7. 2015The Power of Depth for Feedforward Neural Networkscited by 183
  8. 2018Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernelcited by 131
  9. 2019A Function Space View of Bounded Norm Infinite Width ReLU Nets: The Multivariate Casecited by 114
  10. 2019How do infinite width bounded norm networks look in function space?cited by 106
  11. 2019On the Power and Limitations of Random Features for Understanding Neural Networkscited by 103
  12. 2016Global Optimality of Local Search for Low Rank Matrix Recoverycited by 82
21 more
  1. 2020Learning Parities with Neural Networkscited by 78
  2. 2018Gradient Descent Quantizes ReLU Network Featurescited by 61
  3. 2016Depth-Width Tradeoffs in Approximating Natural Functions with Neural Networkscited by 59
  4. 2020A law of robustness for two-layers neural networkscited by 55
  5. 2020Convex Geometry and Duality of Over-parameterized Neural Networkscited by 49
  6. 2021Characterization of the Variation Spaces Corresponding to Shallow Neural Networkscited by 48
  7. 2017Depth Separation for Neural Networkscited by 46
  8. 2019Gradient Dynamics of Shallow Univariate ReLU Networkscited by 46
  9. 2021Quantifying the Benefit of Using Differentiable Learning over Tangent Kernelscited by 41
  10. 2021Near-Minimax Optimal Estimation With Shallow ReLU Neural Networkscited by 41
  11. 2022Random Feature Amplification: Feature Learning and Generalization in Neural Networkscited by 35
  12. 2019Depth Separations in Neural Networks: What is Actually Being Separated?cited by 27
  13. 2016Risk Bounds for High-dimensional Ridge Function Combinations Including Neural Networkscited by 26
  14. 2020Poly-time universality and limitations of deep learningcited by 23
  15. 2019Finding the Needle in the Haystack with Convolutions: on the benefits of architectural biascited by 19
  16. 2021Gradient Descent on Infinitely Wide Neural Networks: Global Convergence and Generalizationcited by 16
  17. 2021Mean-field Analysis of Piecewise Linear Solutions for Wide ReLU Networkscited by 13
  18. 2020Sparsest Piecewise-Linear Regression of One-Dimensional Datacited by 12
  19. 2022Deep Learning meets Nonparametric Regression: Are Weight-Decayed DNNs Locally Adaptive?cited by 10
  20. 2021Ridgeless Interpolation with Shallow ReLU Networks in 1D is Nearest Neighbor Curvature Extrapolation and Provably Generalizes on Lipschitz Functionscited by 9
  21. 2022Astral Space: Convex Analysis at Infinitycited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 2; 2025: 0; 2026: 1.

5 papers.

  1. 2024Emergence in non-neural models: grokking modular arithmetic via average gradient outer productcited by 28
  2. 2023Penalising the biases in norm regularisation enforces sparsitycited by 19
  3. 2023ReLU Neural Networks with Linear Layers are Biased Towards Single- and Multi-Index Modelscited by 8
  4. 2024The Effects of Multi-Task Learning on ReLU Neural Network Functionscited by 0
  5. 2026Non-asymptotic implicit bias of logistic regression at early-stage gradient descent dynamicscited by 0