mlsanityData to 7 Oct 2026

Early-stopped neural networks are consistent

arXiv 2106.05932 ↗First posted 10 Jun 2021cs.LG stat.MLcited by 34

Not on the map. Why?

Abstract (arXiv)

This work studies the behavior of shallow ReLU networks trained with the logistic loss via gradient descent on binary classification data where the underlying data distribution is general, and the (optimal) Bayes risk is not necessarily zero. In this setting, it is shown that gradient descent with early stopping achieves population risk arbitrarily close to optimal in terms of not just logistic and misclassification losses, but also in terms of calibration, meaning the sigmoid mapping of its outputs approximates the true underlying conditional distribution arbitrarily finely. Moreover, the necessary iteration, sample, and architectural complexities of this analysis all scale naturally with a certain complexity measure of the true conditional model. Lastly, while it is not shown that early stopping is necessary, it is shown that any univariate classifier satisfying a local interpolation property is inconsistent.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Builds on 39 of its 66 references that are in the map. First posted on arXiv 10 Jun 2021.

Builds on

39 of 66 references.

References · 39

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2017On Calibration of Modern Neural Networkscited by 4,860
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  4. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  5. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  6. 2017Spectrally-normalized margin bounds for neural networkscited by 681
  7. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  8. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  9. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  10. 2019Benign Overfitting in Linear Regressioncited by 544
  11. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 527
  12. 2017Nonparametric regression using deep neural networks with ReLU activation functioncited by 431
27 more
  1. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  2. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 339
  3. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 308
  4. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
  5. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 260
  6. 2019Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networkscited by 221
  7. 2019Two models of double descent for weak featurescited by 194
  8. 2019Towards moderate overparameterization: global convergence guarantees for training shallow neural networkscited by 176
  9. 2020Directional convergence and alignment in deep learningcited by 172
  10. 2018Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networkscited by 139
  11. 2019Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networkscited by 120
  12. 2018Overfitting or perfect fitting? Risk bounds for classification and regression rules that interpolatecited by 99
  13. 2019How Much Over-parameterization Is Sufficient to Learn Deep ReLU Networks?cited by 84
  14. 2018Risk and parameter convergence of logistic regressioncited by 81
  15. 2019Quadratic Suffices for Over-parametrization via Matrix Chernoff Boundcited by 58
  16. 2020Gradient descent follows the regularization path for general lossescited by 57
  17. 2020Superpolynomial Lower Bounds for Learning One-Layer Neural Networks using Gradient Descentcited by 57
  18. 2020Learning a Single Neuron with Gradient Methodscited by 50
  19. 2020Agnostic Learning of a Single Neuron with Gradient Descentcited by 46
  20. 2020Distributional Generalization: A New Kind of Generalizationcited by 41
  21. 2021Don't Just Blame Over-parametrization for Over-confidence: Theoretical Analysis of Calibration in Binary Classificationcited by 39
  22. 2020Tight Hardness Results for Training Depth-2 ReLU Networkscited by 35
  23. 2019Neural tangent kernels, transportation mappings, and universal approximationcited by 30
  24. 2020Gradient Methods Never Overfit On Separable Datacited by 29
  25. 2021Parameter-free Stochastic Optimization of Variationally Coherent Functionscited by 27
  26. 2013Boosting with the Logistic Loss is Consistentcited by 3
  27. 2021Achieving Small Test Error in Mildly Overparameterized Neural Networkscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 9; 2023: 7; 2024: 6; 2025: 4; 2026: 3.

34 papers.

Sort:

By area:

  1. 2024A Roadmap to Pluralistic Alignmentcited by 175
  2. 2021Assessing Generalization of SGD via Disagreementcited by 116
  3. 2022An Information-theoretic Approach to Prompt Engineering Without Ground Truth Labelscited by 88
  4. 2022Benign Overfitting without Linearity: Neural Network Classifiers Trained by Gradient Descent for Noisy Linear Datacited by 74
  5. 2022Benign, Tempered, or Catastrophic: A Taxonomy of Overfittingcited by 36
  6. 2022Random Feature Amplification: Feature Learning and Generalization in Neural Networkscited by 33
  7. 2025Value Profiles for Encoding Human Variationcited by 29
  8. 2023Mind the spikes: Benign overfitting of kernels and neural networks in fixed dimensioncited by 26
  9. 2021Stability & Generalisation of Gradient Descent for Shallow Neural Networks without the Neural Tangent Kernelcited by 25
  10. 2022Stability and Generalization Analysis of Gradient Methods for Shallow Neural Networkscited by 21
  11. 2022The Spectral Bias of Polynomial Neural Networkscited by 19
  12. 2022Automatic Data Augmentation via Invariance-Constrained Learningcited by 17
22 more
  1. 2023Classification with Deep Neural Networks and Logistic Losscited by 16
  2. 2024Simplicity bias and optimization threshold in two-layer ReLU networkscited by 11
  3. 2025Benefits of Early Stopping in Gradient Descent for Overparameterized Logistic Regressioncited by 9
  4. 2021On the Role of Optimization in Double Descent: A Least Squares Studycited by 7
  5. 2024Near-Interpolators: Rapid Norm Growth and the Trade-Off between Interpolation and Generalizationcited by 7
  6. 2023Connecting NTK and NNGP: A Unified Theoretical Framework for Wide Neural Network Learning Dynamicscited by 6
  7. 2024Transfer Learning Beyond Bounded Density Ratioscited by 6
  8. 2021Nonparametric Regression with Shallow Overparameterized Neural Networks Trained by GD with Early Stoppingcited by 5
  9. 2023Penalized Deep Partially Linear Cox Models with Application to CT Scans of Lung Cancer Patientscited by 4
  10. 2025zScore: A Universal Decentralised Reputation System for the Blockchain Economycited by 4
  11. 2022Towards Data-Algorithm Dependent Generalization: a Case Study on Overparameterized Linear Regressioncited by 3
  12. 2023On Emergence of Clean-Priority Learning in Early Stopped Neural Networkscited by 3
  13. 2022Agnostic Learnability of Halfspaces via Logistic Losscited by 2
  14. 2023On Achieving Optimal Adversarial Test Errorcited by 1
  15. 2023Wrapper Boxes: Faithful Attribution of Model Predictions to Training Datacited by 1
  16. 2024EdgeSync: Faster Edge-model Updating via Adaptive Continuous Learning for Video Data Driftcited by 1
  17. 2026On damage of interpolation to adversarial robustness in regressioncited by 1
  18. 2021Explaining generalization in deep learning: progress and fundamental limitscited by 0
  19. 2024Accelerating Recommender Model Training by Dynamically Skipping Stale Embeddingscited by 0
  20. 2025GradES: Significantly Faster Training in Transformers with Gradient-Based Early Stoppingcited by 0
  21. 2026Don't stop me now: How Validation Criteria Affect Checkpoint Selection and Early Stoppingcited by 0
  22. 2026Deep Neural Networks for Doubly Robust Estimation with Nonprobability Survey Samplescited by 0