mlsanityData to 14 Oct 2026

The Directional Bias Helps Stochastic Gradient Descent to Generalize in Kernel Regression Models

arXiv 2205.00061 ↗First posted 29 Apr 2022stat.ML cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

We study the Stochastic Gradient Descent (SGD) algorithm in nonparametric statistics: kernel regression in particular. The directional bias property of SGD, which is known in the linear regression setting, is generalized to the kernel regression. More specifically, we prove that SGD with moderate and annealing step-size converges along the direction of the eigenvector that corresponds to the largest eigenvalue of the Gram matrix. In addition, the Gradient Descent (GD) with a moderate or small step-size converges along the direction that corresponds to the smallest eigenvalue. These facts are referred to as the directional bias properties; they may interpret how an SGD-computed estimator has a potentially smaller generalization error than a GD-computed estimator. The application of our theory is demonstrated by simulation studies and a case study that is based on the FashionMNIST dataset.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 16 of its 34 references that are in the map. First posted on arXiv 29 Apr 2022.

Builds on

16 of 34 references.

References · 16

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  3. 2021Deep learning: a statistical viewpointcited by 240
  4. 2018Just Interpolate: Kernel "Ridgeless" Regression Can Generalizecited by 190
  5. 2018To understand deep learning we need to understand kernel learningcited by 187
  6. 2019Towards Understanding the Spectral Bias of Deep Learningcited by 174
  7. 2017The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learningcited by 150
  8. 2019Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like processcited by 123
  9. 2021Dive into Deep Learningcited by 111
  10. 2019Implicit Regularization for Optimal Sparse Recoverycited by 80
  11. 2018Statistical Optimality of Stochastic Gradient Descent on Hard Learning Problems through Multiple Passescited by 69
  12. 2021What Kinds of Functions do Deep Neural Networks Learn? Insights from Variational Spline Theorycited by 60
4 more
  1. 2020The Implicit Regularization of Stochastic Gradient Flow for Least Squarescited by 59
  2. 2019Exact expressions for double descent and implicit regularization via surrogate random designcited by 35
  3. 2020Direction Matters: On the Implicit Bias of Stochastic Gradient Descent with Moderate Learning Ratecited by 12
  4. 2021Grouped Variable Selection with Discrete Optimization: Computational and Statistical Perspectivescited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Complex Stochastic Gradient Descent and Directional Bias in Reproducing Kernel Hilbert Spacescited by 0