mlsanityData to 7 Oct 2026

Implicit Regularization in ReLU Networks with the Square Loss

arXiv 2012.05156 ↗First posted 9 Dec 2020cs.LG stat.MLcited by 45

Not on the map. Why?

Abstract (arXiv)

Understanding the implicit regularization (or implicit bias) of gradient descent has recently been a very active research area. However, the implicit regularization in nonlinear neural networks is still poorly understood, especially for regression losses such as the square loss. Perhaps surprisingly, we prove that even for a single ReLU neuron, it is impossible to characterize the implicit regularization with the square loss by any explicit function of the model parameters (although on the positive side, we show it can be characterized approximately). For one hidden-layer networks, we prove a similar result, where in general it is impossible to characterize implicit regularization properties in this manner, except for the "balancedness" property identified in Du et al. [2018]. Our results suggest that a more general framework than the one considered so far may be needed to understand implicit regularization for nonlinear predictors, and provides some clues on what this framework should be.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). First posted on arXiv 9 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 12; 2023: 12; 2024: 12; 2025: 1; 2026: 2.

45 papers.

Sort:

  1. 2021Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gaugescited by 1,131
  2. 2022Reconstructing Training Data from Trained Neural Networkscited by 129
  3. 2022On the Implicit Bias in Deep-Learning Algorithmscited by 84
  4. 2022Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputscited by 74
  5. 2021On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descentcited by 69
  6. 2022Implicit Regularization Towards Rank Minimization in ReLU Networkscited by 53
  7. 2021Implicit Regularization in Tensor Factorizationcited by 43
  8. 2024Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learningcited by 41
  9. 2021Continuous vs. Discrete Optimization of Deep Neural Networkscited by 34
  10. 2021On Margin Maximization in Linear and ReLU Networkscited by 34
  11. 2024Implicit Bias and Fast Convergence Rates for Self-attentioncited by 31
  12. 2022On Implicit Bias in Overparameterized Bilevel Optimizationcited by 30
33 more
  1. 2024Approaching Deep Learning through the Spectral Dynamics of Weightscited by 29
  2. 2022Implicit Regularization in Hierarchical Tensor Factorization and Deep Convolutional Neural Networkscited by 27
  3. 2024Compressible Dynamics in Deep Overparameterized Low-Rank Learning & Adaptationcited by 26
  4. 2023Implicit regularization in Heavy-ball momentum accelerated stochastic gradient descentcited by 25
  5. 2023Deconstructing Data Reconstruction: Multiclass, Weight Decay and General Lossescited by 24
  6. 2024Implicit Geometry of Next-token Prediction: From Language Sparsity Patterns to Model Representationscited by 24
  7. 2022On the Effective Number of Linear Regions in Shallow Univariate ReLU Networks: Convergence Guarantees and Implicit Biascited by 21
  8. 2023The Law of Parsimony in Gradient Descent for Learning Deep Linear Networkscited by 19
  9. 2024When does compositional structure yield compositional generalization? A kernel theorycited by 19
  10. 2021Learning a Single Neuron with Bias Using Gradient Descentcited by 17
  11. 2023Penalising the biases in norm regularisation enforces sparsitycited by 17
  12. 2023Learning a Neuron by a Shallow ReLU Network: Dynamics and Implicit Bias for Correlated Inputscited by 16
  13. 2023The Implicit Bias of Minima Stability in Multivariate Shallow ReLU Networkscited by 15
  14. 2024NeuZip: Memory-Efficient Training and Inference with Dynamic Compression of Neural Networkscited by 12
  15. 2025The Rich and the Simple: On the Implicit Bias of Adam and SGDcited by 12
  16. 2023On a continuous time model of gradient descent dynamics and instability in deep learningcited by 11
  17. 2024Optimizing Attention with Mirror Descent: Generalized Max-Margin Token Selectioncited by 9
  18. 2022Deep Linear Networks can Benignly Overfit when Shallow Ones Docited by 8
  19. 2023Guided Deep Kernel Learningcited by 6
  20. 2026To Grok Grokking: Provable Grokking in Ridge Regressioncited by 6
  21. 2022Magnitude and Angle Dynamics in Training Single ReLU Neuronscited by 5
  22. 2022From Gradient Flow on Population Loss to Learning with Stochastic Gradient Descentcited by 5
  23. 2024ReLUs Are Sufficient for Learning Implicit Neural Representationscited by 5
  24. 2023Transformed Low-Rank Parameterization Can Help Robust Generalization for Tensor Neural Networkscited by 4
  25. 2023Implicit biases in multitask and continual learning from a backward error analysis perspectivecited by 4
  26. 2024When Are Bias-Free ReLU Networks Effectively Linear Networks?cited by 3
  27. 2023Implicit regularisation in stochastic gradient descent: from single-objective to two-player gamescited by 2
  28. 2024Optimization Insights into Deep Diagonal Linear Networkscited by 2
  29. 2023Mixed Semi-Supervised Generalized-Linear-Regression with Applications to Deep-Learning and Interpolatorscited by 1
  30. 2022Limitation of Characterizing Implicit Regularization by Data-independent Functionscited by 0
  31. 2022Support Vectors and Gradient Dynamics of Single-Neuron ReLU Networkscited by 0
  32. 2024Generalization bounds for regression and classification on adaptive covering input domainscited by 0
  33. 2026Not all solutions are created equal: An analytical dissociation of functional and representational similarity in deep linear neural networkscited by 0