mlsanityData to 7 Oct 2026

Early stopping and non-parametric regression: An optimal data-dependent stopping rule

arXiv 1306.3574 ↗First posted 15 Jun 2013stat.MLcited by 86

Not on the map. Why?

Abstract (arXiv)

The strategy of early stopping is a regularization technique based on choosing a stopping time for an iterative algorithm. Focusing on non-parametric regression in a reproducing kernel Hilbert space, we analyze the early stopping strategy for a form of gradient-descent applied to the least-squares loss function. We propose a data-dependent stopping rule that does not involve hold-out or cross-validation data, and we prove upper bounds on the squared error of the resulting function estimate, measured in either the L2(P) and L2(Pn) norm. These upper bounds lead to minimax-optimal rates for various kernel classes, including Sobolev smoothness classes and other forms of reproducing kernel Hilbert spaces. We show through simulation that our stopping rule compares favorably to two other stopping rules, one based on hold-out data and the other based on Stein's unbiased risk estimate. We also establish a tight connection between our early stopping strategy and the solution path of a kernel ridge regression estimator.

Key facts. Cited by 86 papers in the map (data through 7 Oct 2026). First posted on arXiv 15 Jun 2013.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 12; 2022: 14; 2023: 17; 2024: 14; 2025: 13; 2026: 16.

86 papers.

Sort:

By area:

  1. 2022A Solvable Model of Neural Scaling Lawscited by 101
  2. 2021Label-Imbalanced and Group-Sensitive Classification under Overparameterizationcited by 74
  3. 2024Neural Network-Based Score Estimation in Diffusion Models: Optimization and Generalizationcited by 39
  4. 2024A Statistical Theory of Regularization-Based Continual Learningcited by 32
  5. 2021Implicit Sparse Regularization: The Impact of Depth and Early Stoppingcited by 30
  6. 2023Generalization Ability of Wide Neural Networks on ℝcited by 28
  7. 2023Mind the spikes: Benign overfitting of kernels and neural networks in fixed dimensioncited by 26
  8. 2021The loss landscape of deep linear neural networks: a second-order analysiscited by 24
  9. 2021The Internet of Federated Things (IoFT): A Vision for the Future and In-depth Survey of Data-driven Approaches for Federated Learningcited by 23
  10. 2021The Three Stages of Learning Dynamics in High-Dimensional Kernel Methodscited by 21
  11. 2021A Non-parametric View of FedAvg and FedProx: Beyond Stationary Pointscited by 13
  12. 2021Channel-Wise Early Stopping without a Validation Set via NNK Polytope Interpolationcited by 11
38 more
  1. 2023Optimal Rate of Kernel Regression in Large Dimensionscited by 10
  2. 2024On Regularization via Early Stopping for Least Squares Regressioncited by 10
  3. 2022Exponential Tail Local Rademacher Complexity Risk Bounds Without the Bernstein Conditioncited by 9
  4. 2023Strong inductive biases provably prevent harmless interpolationcited by 9
  5. 2025Spurious Correlations in High Dimensional Regression: The Roles of Regularization, Simplicity Bias and Over-Parameterizationcited by 9
  6. 2022Risk Bounds of Multi-Pass SGD for Least Squares in the Interpolation Regimecited by 8
  7. 2022Data adaptive RKHS Tikhonov regularization for learning kernels in operatorscited by 8
  8. 2022Graph Neural Networks for Low-Energy Event Classification & Reconstruction in IceCubecited by 8
  9. 2021Click-through Rate Prediction with Auto-Quantized Contrastive Learningcited by 7
  10. 2024Random features models: a way to study the success of naive imputationcited by 7
  11. 2023Random Smoothing Regularization in Kernel Gradient Descent Learningcited by 6
  12. 2023Adversarial Contextual Bandits Go Kernelizedcited by 6
  13. 2024The Optimality of (Accelerated) SGD for High-Dimensional Quadratic Optimizationcited by 6
  14. 2021Nonparametric Regression with Shallow Overparameterized Neural Networks Trained by GD with Early Stoppingcited by 5
  15. 2021Comparing Classes of Estimators: When does Gradient Descent Beat Ridge Regression in Linear Models?cited by 5
  16. 2022A Neural Phillips Curve and a Deep Output Gapcited by 5
  17. 2022Lazy Estimation of Variable Importance for Large Neural Networkscited by 5
  18. 2022Learning Lipschitz Functions by GD-trained Shallow Overparameterized ReLU Neural Networkscited by 5
  19. 2023Optimality of Robust Online Learningcited by 5
  20. 2023Generalization Ability of Wide Residual Networkscited by 5
  21. 2023Classical Out-of-Distribution Detection Methods Benchmark in Text Classification Taskscited by 5
  22. 2021From inexact optimization to learning via gradient concentrationcited by 4
  23. 2021Regularizing Transformers With Deep Probabilistic Layerscited by 4
  24. 2022Behind the Scenes of Gradient Descent: A Trajectory Analysis via Basis Function Decompositioncited by 4
  25. 2023Fractional dynamics foster deep learning of COPD stage predictioncited by 4
  26. 2024Failures and Successes of Cross-Validation for Early-Stopped Gradient Descentcited by 4
  27. 2025Design Considerations in Offline Preference-based RLcited by 4
  28. 2022Towards Data-Algorithm Dependent Generalization: a Case Study on Overparameterized Linear Regressioncited by 3
  29. 2023CROP: Towards Distributional-Shift Robust Reinforcement Learning using Compact Reshaped Observation Processingcited by 3
  30. 2023Adaptive Parameter Selection for Kernel Ridge Regressioncited by 3
  31. 2025Learning Curves of Stochastic Gradient Descent in Kernel Regressioncited by 3
  32. 2025Spectral Algorithms in Misspecified Regression: Convergence under Covariate Shiftcited by 3
  33. 2023Quantifying Overfitting: Introducing the Overfitting Indexcited by 2
  34. 2023Adaptive Distributed Kernel Ridge Regression: A Feasible Distributed Learning Scheme for Data Siloscited by 2
  35. 2023From Reactive to Proactive Volatility Modeling with Hemisphere Neural Networkscited by 2
  36. 2024On the Rashomon ratio of infinite hypothesis setscited by 2
  37. 2024Gradient Descent Finds Over-Parameterized Neural Networks with Sharp Generalization for Nonparametric Regressioncited by 2
  38. 2024Reliable and scalable variable importance estimation via warm-start and early stoppingcited by 2

50 most cited of 86 All 86 (JSON)