mlsanityData to 7 Oct 2026

Limitation of Characterizing Implicit Regularization by Data-independent Functions

arXiv 2201.12198 ↗First posted 28 Jan 2022cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

In recent years, understanding the implicit regularization of neural networks (NNs) has become a central task in deep learning theory. However, implicit regularization is itself not completely defined and well understood. In this work, we attempt to mathematically define and study implicit regularization. Importantly, we explore the limitations of a common approach to characterizing implicit regularization using data-independent functions. We propose two dynamical mechanisms, i.e., Two-point and One-point Overlapping mechanisms, based on which we provide two recipes for producing classes of one-hidden-neuron NNs that provably cannot be fully characterized by a type of or all data-independent functions. Following the previous works, our results further emphasize the profound data dependency of implicit regularization in general, inspiring us to study in detail the data dependency of NN implicit regularization in the future.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 27 of its 52 references that are in the map. First posted on arXiv 28 Jan 2022.

Builds on

27 of 52 references.

References · 27

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  3. 2017A Closer Look at Memorization in Deep Networkscited by 936
  4. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  5. 2019Implicit Regularization in Deep Matrix Factorizationcited by 402
  6. 2019Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networkscited by 360
  7. 2017Implicit Regularization in Matrix Factorizationcited by 318
  8. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 302
  9. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 280
  10. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
  11. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  12. 2019SGD on Neural Networks Learns Functions of Increasing Complexitycited by 203
15 more
  1. 2019Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limitcited by 187
  2. 2018Training behavior of deep neural network in frequency domaincited by 174
  3. 2020Implicit Regularization in Deep Learning May Not Be Explainable by Normscited by 118
  4. 2017Regularization for Deep Learning: A Taxonomycited by 91
  5. 2022Overview frequency principle/spectral bias in deep learningcited by 68
  6. 2020Phase diagram for two-layer ReLU neural networks at infinite-width limitcited by 65
  7. 2019The Implicit Bias of Depth: How Incremental Learning Drives Generalizationcited by 64
  8. 2020Implicit Regularization in ReLU Networks with the Square Losscited by 45
  9. 2020Gradient Descent for Deep Matrix Factorization: Dynamics and Implicit Bias towards Low Rankcited by 42
  10. 2019A type of generalization error induced by initialization in deep neural networkscited by 33
  11. 2020Deep frequency principle towards understanding why deeper learning is fastercited by 30
  12. 2019Quantifying the generalization error in deep learning in terms of data distribution and neural network smoothnesscited by 22
  13. 2020Can Implicit Bias Explain Generalization? Stochastic Convex Optimization as a Case Studycited by 16
  14. 2020On the exact computation of linear frequency principle dynamics and its generalizationcited by 16
  15. 2021Linear Frequency Principle Model to Understand the Absence of Overfitting in Neural Networkscited by 14

Built on by

Not cited yet.