mlsanityData to 14 Oct 2026

Disparity Between Batches as a Signal for Early Stopping

arXiv 2107.06665 ↗First posted 14 Jul 2021cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

We propose a metric for evaluating the generalization ability of deep neural networks trained with mini-batch gradient descent. Our metric, called gradient disparity, is the ℓ2 norm distance between the gradient vectors of two mini-batches drawn from the training set. It is derived from a probabilistic upper bound on the difference between the classification errors over a given mini-batch, when the network is trained on this mini-batch and when the network is trained on another mini-batch of points sampled from the same dataset. We empirically show that gradient disparity is a very promising early-stopping criterion (i) when data is limited, as it uses all the samples for training and (ii) when available data has noisy labels, as it signals overfitting better than the validation data. Furthermore, we show in a wide range of experimental settings that gradient disparity is strongly related to the generalization error between the training and test sets, and that it is also very informative about the level of label noise.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 35 of its 65 references that are in the map. First posted on arXiv 14 Jul 2021.

Builds on

35 of 65 references.

References · 35

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  4. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  5. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  6. 2016An overview of gradient descent optimization algorithmscited by 1,033
  7. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
  8. 2017Exploring Generalization in Deep Learningcited by 600
  9. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 533
  10. 2018Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interferencecited by 521
  11. 2015Recent Advances in Convolutional Neural Networkscited by 360
  12. 2017A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networkscited by 314
23 more
  1. 2019Uniform convergence may be unable to explain generalization in deep learningcited by 224
  2. 2019Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networkscited by 159
  3. 2020The Break-Even Point on Optimization Trajectories of Deep Neural Networkscited by 150
  4. 2020Evaluation of Neural Architectures Trained with Square Loss vs Cross-Entropy in Classification Taskscited by 138
  5. 2019Information-Theoretic Generalization Bounds for SGLD via Data-Dependent Estimatescited by 119
  6. 2018A Survey on Data Collection for Machine Learning: a Big Data -- AI Integration Perspectivecited by 112
  7. 2020Sharpened Generalization Bounds based on Conditional Mutual Information and an Application to Noisy, Iterative Algorithmscited by 89
  8. 2020On the role of data in PAC-Bayes boundscited by 64
  9. 2019Stiffness: A New Perspective on Generalization in Neural Networkscited by 46
  10. 2019The Impact of Neural Network Overparameterization on Gradient Confusion and Stochastic Gradient Descentcited by 46
  11. 2020Interference and Generalization in Temporal Difference Learningcited by 42
  12. 2020Toward Better Generalization Bounds with Locally Elastic Stabilitycited by 42
  13. 2019How does Early Stopping Help Generalization against Label Noise?cited by 41
  14. 2020Coherent Gradients: An Approach to Understanding Generalization in Gradient Descent-based Optimizationcited by 41
  15. 2020Early Stopping in Deep Networks: Double Descent and How to Eliminate itcited by 41
  16. 2019The intriguing role of module criticality in the generalization of deep networkscited by 40
  17. 2020Understanding Why Neural Networks Generalize Well Through GSNR of Parameterscited by 40
  18. 2017Early Stopping without a Validation Setcited by 39
  19. 2019The Local Elasticity of Neural Networkscited by 37
  20. 2020Rethinking Generalization of Neural Models: A Named Entity Recognition Case Studycited by 35
  21. 2020Extreme Memorization via Scale of Initializationcited by 18
  22. 2020The Impact of the Mini-batch Size on the Variance of Gradients in Stochastic Gradient Descentcited by 16
  23. 2021Optimization Variance: Exploring Generalization Properties of DNNscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 2; 2024: 2; 2025: 2; 2026: 1.

9 papers.

  1. 2024A Layer Selection Approach to Test Time Adaptationcited by 7
  2. 2022Towards Data-Algorithm Dependent Generalization: a Case Study on Overparameterized Linear Regressioncited by 3
  3. 2022Intersection of Parallels as an Early Stopping Criterioncited by 3
  4. 2023Addressing caveats of neural persistence with deep graph persistencecited by 1
  5. 2025Gradient-Weight Alignment as a Train-Time Proxy for Generalization in Classification Taskscited by 1
  6. 2023Double Descent of Discrepancy: A Task-, Data-, and Model-Agnostic Phenomenoncited by 0
  7. 2024Twin: Tuning Learning Rate and Weight Decay of Deep Homogeneous Classifiers without Validationcited by 0
  8. 2025GRADSTOP: Early Stopping of Gradient Descent via Posterior Samplingcited by 0
  9. 2026Information Hidden in Gradients of Regression with Target Noisecited by 0