mlsanityData to 14 Oct 2026

Assessing Generalization of SGD via Disagreement

arXiv 2106.13799 ↗First posted 25 Jun 2021cs.LG cs.AI stat.MLcited by 117

Not on the map. Why?

Abstract (arXiv)

We empirically show that the test error of deep networks can be estimated by simply training the same architecture on the same training set but with a different run of Stochastic Gradient Descent (SGD), and measuring the disagreement rate between the two networks on unlabeled test data. This builds on -- and is a stronger version of -- the observation in Nakkiran & Bansal '20, which requires the second run to be on an altogether fresh training set. We further theoretically show that this peculiar phenomenon arises from the well-calibrated nature of ensembles of SGD-trained models. This finding not only provides a simple empirical measure to directly predict the test error using unlabeled test data, but also establishes a new conceptual connection between generalization and calibration.

Key facts. Cited by 117 papers in the map (data through 14 Oct 2026). Builds on 48 of its 71 references that are in the map. First posted on arXiv 25 Jun 2021.

Builds on

48 of 71 references.

References · 48

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2017On Calibration of Modern Neural Networkscited by 4,960
  3. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  4. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  5. 2017Deeper, Broader and Artier Domain Generalizationcited by 986
  6. 2017A Closer Look at Memorization in Deep Networkscited by 942
  7. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
  8. 2017Spectrally-normalized margin bounds for neural networkscited by 686
  9. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 672
  10. 2017Exploring Generalization in Deep Learningcited by 600
  11. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 533
  12. 2019Fantastic Generalization Measures and Where to Find Themcited by 513
36 more
  1. 2019Measuring Calibration in Deep Learningcited by 446
  2. 2019Deep Ensembles: A Loss Landscape Perspectivecited by 443
  3. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 342
  4. 2017A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networkscited by 314
  5. 2019Verified Uncertainty Calibrationcited by 309
  6. 2020Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learningcited by 297
  7. 2019Uniform convergence may be unable to explain generalization in deep learningcited by 224
  8. 2019SGD on Neural Networks Learns Functions of Increasing Complexitycited by 204
  9. 2018To understand deep learning we need to understand kernel learningcited by 187
  10. 2019Evaluating model calibration in classificationcited by 168
  11. 2018Predicting the Generalization Gap in Deep Networks with Margin Distributionscited by 113
  12. 2020Predicting Neural Network Accuracy from Weightscited by 110
  13. 2019Calibration tests in multi-class classification: A unifying frameworkcited by 79
  14. 2020Distribution-free binary classification: prediction sets, confidence intervals and calibrationcited by 79
  15. 2020In Search of Robust Measures of Generalizationcited by 79
  16. 2018A Modern Take on the Bias-Variance Tradeoff in Neural Networkscited by 65
  17. 2020Moment Multicalibration for Uncertainty Estimationcited by 63
  18. 2020Estimating Generalization under Distribution Shifts via Domain-Invariant Representationscited by 56
  19. 2019Generalization in Deep Networks: The Role of Distance from Initializationcited by 51
  20. 2019Deterministic PAC-Bayesian generalization bounds for deep networks via generalizing noise-resiliencecited by 51
  21. 2019In Defense of Uniform Convergence: Generalization via derandomization with an application to interpolating predictorscited by 44
  22. 2020Distributional Generalization: A New Kind of Generalizationcited by 42
  23. 2020NeurIPS 2020 Competition: Predicting Generalization in Deep Learningcited by 41
  24. 2021Don't Just Blame Over-parametrization for Over-confidence: Theoretical Analysis of Calibration in Binary Classificationcited by 39
  25. 2021Early-stopped neural networks are consistentcited by 34
  26. 2020Representation Based Complexity Measures for Predicting Generalization in Deep Learningcited by 29
  27. 2021On the Reproducibility of Neural Network Predictionscited by 29
  28. 2017Estimating Accuracy from Unlabeled Data: A Probabilistic Logic Approachcited by 27
  29. 2020Sample Complexity of Uniform Convergence for Multicalibrationcited by 24
  30. 2020On Uniform Convergence and Low-Norm Interpolation Learningcited by 24
  31. 2021RATT: Leveraging Unlabeled Data to Guarantee Generalizationcited by 20
  32. 2019Towards Task and Architecture-Independent Generalization Gap Predictorscited by 18
  33. 2014Estimating the Accuracies of Multiple Classifiers Without Labeled Datacited by 17
  34. 2016Unsupervised Risk Estimation Using Only Conditional Independence Structurecited by 17
  35. 2021Churn Reduction via Distillationcited by 17
  36. 2021Should Ensemble Members Be Calibrated?cited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 4; 2022: 23; 2023: 34; 2024: 27; 2025: 18; 2026: 11.

117 papers.

Sort:

By area:

  1. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 471
  2. 2022Datamodels: Predicting Predictions from Training Datacited by 189
  3. 2023Large Language Model Routing with Benchmark Datasetscited by 172
  4. 2022Leveraging Unlabeled Data to Predict Out-of-Distribution Performancecited by 145
  5. 2022Agreement-on-the-Line: Predicting the Performance of Neural Networks under Distribution Shiftcited by 94
  6. 2021Classifier Calibration: A survey on how to assess and improve predicted class probabilitiescited by 73
  7. 2024Exponential Moving Average of Weights in Deep Learning: Dynamics and Benefitscited by 71
  8. 2021Detecting Errors and Estimating Accuracy on Unlabeled Data with Self-training Ensemblescited by 54
  9. 2023Towards Last-layer Retraining for Group Robustness with Fewer Annotationscited by 54
  10. 2022Predicting Out-of-Distribution Error with the Projection Normcited by 50
  11. 2022ModelDiff: A Framework for Comparing Learning Algorithmscited by 30
  12. 2023RLSbench: Domain Adaptation Under Relaxed Label Shiftcited by 30
38 more
  1. 2024The Entropy Enigma: Success and Failure of Entropy Minimizationcited by 24
  2. 2022On the Strong Correlation Between Model Invariance and Generalizationcited by 23
  3. 2024What Do Learning Dynamics Reveal About Generalization in LLM Reasoning?cited by 23
  4. 2023On the Variance of Neural Network Training with respect to Test Sets and Distributionscited by 22
  5. 2023Multimodal Learning Without Labeled Multimodal Data: Guarantees and Applicationscited by 21
  6. 2024A Survey on Evaluation of Out-of-Distribution Generalizationcited by 21
  7. 2022Selective Prediction via Training Dynamicscited by 20
  8. 2023Characterizing Out-of-Distribution Error via Optimal Transportcited by 20
  9. 2023GNNEvaluator: Evaluating GNN Performance On Unseen Graphs Without Labelscited by 17
  10. 2024Energy-based Automated Model Evaluationcited by 17
  11. 2021A Framework for Cluster and Classifier Evaluation in the Absence of Reference Labelscited by 16
  12. 2022A Note on "Assessing Generalization of SGD via Disagreement"cited by 16
  13. 2022Detecting Label Errors by using Pre-Trained Language Modelscited by 16
  14. 2023(Almost) Provable Error Bounds Under Distribution Shift via Disagreement Discrepancycited by 16
  15. 202494% on CIFAR-10 in 3.29 Seconds on a Single GPUcited by 16
  16. 2023On the Importance of Feature Separability in Predicting Out-Of-Distribution Errorcited by 15
  17. 2024A Framework for Efficient Model Evaluation through Stratification, Sampling, and Estimationcited by 14
  18. 2023Predicting Out-of-Distribution Error with Confidence Optimal Transportcited by 13
  19. 2023Demystifying Disagreement-on-the-Line in High Dimensionscited by 12
  20. 2023LOVM: Language-Only Vision Model Selectioncited by 12
  21. 2024LCA-on-the-Line: Benchmarking Out-of-Distribution Generalization with Class Taxonomiescited by 12
  22. 2022Understanding new tasks through the lens of training data via exponential tiltingcited by 11
  23. 2023Test-Time Adaptation Induces Stronger Accuracy and Agreement-on-the-Linecited by 11
  24. 2025Monitoring Risks in Test-Time Adaptationcited by 11
  25. 2022Evaluating Robustness to Dataset Shift via Parametric Robustness Setscited by 10
  26. 2022Monitoring Model Deterioration with Explainable Uncertainty Estimation via Non-parametric Bootstrapcited by 9
  27. 2022Task Discovery: Finding the Tasks that Neural Networks Generalize oncited by 9
  28. 2024Estimating Model Performance Under Covariate Shift Without Labelscited by 9
  29. 2024AETTA: Label-Free Accuracy Estimation for Test-Time Adaptationcited by 9
  30. 2025Predicting the Performance of Black-box LLMs through Follow-up Queriescited by 9
  31. 2023A Bias-Variance-Covariance Decomposition of Kernel Scores for Generative Modelscited by 8
  32. 2024MANO: Exploiting Matrix Norm for Unsupervised Accuracy Estimation Under Distribution Shiftscited by 8
  33. 2024Confidence-based Estimators for Predictive Performance in Model Monitoringcited by 8
  34. 2022Understanding Deep Learning via Decision Boundarycited by 7
  35. 2023Selecting which Dense Retriever to use for Zero-Shot Searchcited by 7
  36. 2023Leveraging Ensemble Diversity for Robust Self-Training in the Presence of Sample Selection Biascited by 7
  37. 2024Advancing Deep Active Learning & Data Subset Selection: Unifying Principles with Information-Theory Intuitionscited by 7
  38. 2024Predicting the Performance of Foundation Models via Agreement-on-the-Linecited by 7

50 most cited of 117 All 117 (JSON)