mlsanityData to 14 Oct 2026

Penalising the biases in norm regularisation enforces sparsity

arXiv 2303.01353 ↗First posted 2 Mar 2023stat.ML cs.LGcited by 19

Not on the map. Why?

Abstract (arXiv)

Controlling the parameters' norm often yields good generalisation when training neural networks. Beyond simple intuitions, the relation between regularising parameters' norm and obtained estimators remains theoretically misunderstood. For one hidden ReLU layer networks with unidimensional data, this work shows the parameters' norm required to represent a function is given by the total variation of its second derivative, weighted by a √1+x2 factor. Notably, this weighting factor disappears when the norm of bias terms is not regularised. The presence of this additional weighting factor is of utmost significance as it is shown to enforce the uniqueness and sparsity (in the number of kinks) of the minimal norm interpolator. Conversely, omitting the bias' norm allows for non-sparse solutions. Penalising the bias terms in the regularisation, either explicitly or implicitly, thus leads to sparse estimators.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Builds on 24 of its 43 references that are in the map. First posted on arXiv 2 Mar 2023.

Builds on

24 of 43 references.

References · 24

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
  3. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
  4. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 342
  5. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 331
  6. 2017Implicit Regularization in Matrix Factorizationcited by 322
  7. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 320
  8. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 268
  9. 2020Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learningcited by 150
  10. 2020Implicit Regularization in Deep Learning May Not Be Explainable by Normscited by 119
  11. 2019A Function Space View of Bounded Norm Infinite Width ReLU Nets: The Multivariate Casecited by 114
  12. 2019How do infinite width bounded norm networks look in function space?cited by 106
12 more
  1. 2022Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputscited by 76
  2. 2020Representation formulas and pointwise properties for Barron functionscited by 62
  3. 2020Implicit Regularization in ReLU Networks with the Square Losscited by 45
  4. 2020The Hidden Convex Optimization Landscape of Two-Layer ReLU Neural Networks: an Exact Characterization of the Optimal Solutionscited by 30
  5. 2022Regression as Classification: Influence of Task Formulation on Neural Network Featurescited by 29
  6. 2022On the Effective Number of Linear Regions in Shallow Univariate ReLU Networks: Convergence Guarantees and Implicit Biascited by 21
  7. 2023Deep Learning Meets Sparse Regularization: A Signal Processing Perspectivecited by 21
  8. 2020Complexity Measures for Neural Networks with General Activation Functions Using Path-based Normscited by 19
  9. 2021Mean-field Analysis of Piecewise Linear Solutions for Wide ReLU Networkscited by 13
  10. 2020Sparsest Piecewise-Linear Regression of One-Dimensional Datacited by 12
  11. 2022Intrinsic dimensionality and generalization properties of the R-norm inductive biascited by 5
  12. 2015Super-Resolution of Point Sources via Convex Programmingcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 4; 2025: 5; 2026: 4.

19 papers.

Sort:

By area:

  1. 2025Taking a Big Step: Large Learning Rates in Denoising Score Matching Prevent Memorizationcited by 17
  2. 2023From Tempered to Benign Overfitting in ReLU Neural Networkscited by 16
  3. 2023Learning a Neuron by a Shallow ReLU Network: Dynamics and Implicit Bias for Correlated Inputscited by 16
  4. 2025A Theoretical Framework for Grokking: Interpolation followed by Riemannian Norm Minimisationcited by 13
  5. 2023Noisy Interpolation Learning with Shallow Univariate ReLU Networkscited by 11
  6. 2024Simplicity bias and optimization threshold in two-layer ReLU networkscited by 11
  7. 2023How do Minimum-Norm Shallow Denoisers Look in Function Space?cited by 9
  8. 2023ReLU Neural Networks with Linear Layers are Biased Towards Single- and Multi-Index Modelscited by 8
  9. 2023Minimum norm interpolation by perceptra: Explicit regularization and implicit biascited by 3
  10. 2024Depth Separation in Norm-Bounded Infinite-Width Neural Networkscited by 3
  11. 2024Solving the Poisson Equation with Dirichlet data by shallow ReLUα-networks: A regularity and approximation perspectivecited by 2
  12. 2025Favorability of Loss Landscape with Weight Decay Requires Both Large Overparametrization and Initializationcited by 1
7 more
  1. 2026On damage of interpolation to adversarial robustness in regressioncited by 1
  2. 2026Elliptic Regularity Theory in Barron Spaces and Applications to the Deep Ritz Methodcited by 1
  3. 2024The Effects of Multi-Task Learning on ReLU Neural Network Functionscited by 0
  4. 2025Global Minimizers of ℓp-Regularized Objectives Yield the Sparsest ReLU Neural Networkscited by 0
  5. 2025A Recovery Guarantee for Sparse Neural Networkscited by 0
  6. 2026Mildly Overparameterized ReLU Networks on Orthogonal Data: Incremental Learning and Implicit Biascited by 0
  7. 2026Minimal-Norm Univariate Two-Layer ReLU Classification: Exact Solutions and Global Optimality with Skip Connectionscited by 0