mlsanityData to 7 Oct 2026

A Note on "Assessing Generalization of SGD via Disagreement"

arXiv 2202.01851 ↗First posted 3 Feb 2022cs.LG cs.AIcited by 16

Not on the map. Why?

Abstract (arXiv)

Several recent works find empirically that the average test error of deep neural networks can be estimated via the prediction disagreement of models, which does not require labels. In particular, Jiang et al. (2022) show for the disagreement between two separately trained networks that this `Generalization Disagreement Equality' follows from the well-calibrated nature of deep ensembles under the notion of a proposed `class-aggregated calibration.' In this reproduction, we show that the suggested theory might be impractical because a deep ensemble's calibration can deteriorate as prediction disagreement increases, which is precisely when the coupling of test error and disagreement is of interest, while labels are needed to estimate the calibration on new datasets. Further, we simplify the theoretical statements and proofs, showing them to be straightforward within a probabilistic context, unlike the original hypothesis space view employed by Jiang et al. (2022).

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 29 of its 46 references that are in the map. First posted on arXiv 3 Feb 2022.

Builds on

29 of 46 references.

References · 29

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2017On Calibration of Modern Neural Networkscited by 4,860
  5. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  6. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  7. 2022A ConvNet for the 2020scited by 3,545
  8. 2016Wide Residual Networkscited by 3,057
  9. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
  10. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  11. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,335
  12. 2018A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attackscited by 1,472
17 more
  1. 2017Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networkscited by 1,325
  2. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  3. 2017Deeper, Broader and Artier Domain Generalizationcited by 983
  4. 2017Deep Bayesian Active Learning with Image Datacited by 812
  5. 2020Bayesian Deep Learning and a Probabilistic Perspective of Generalizationcited by 480
  6. 2019Measuring Calibration in Deep Learningcited by 443
  7. 2018Bag of Tricks for Image Classification with Convolutional Neural Networkscited by 430
  8. 2019BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learningcited by 428
  9. 2018Understanding Measures of Uncertainty for Adversarial Example Detectioncited by 177
  10. 2022Leveraging Unlabeled Data to Predict Out-of-Distribution Performancecited by 141
  11. 2021Assessing Generalization of SGD via Disagreementcited by 116
  12. 2021Detecting Errors and Estimating Accuracy on Unlabeled Data with Self-training Ensemblescited by 54
  13. 2021DOCTOR: A Simple Method for Detecting Misclassification Errorscited by 52
  14. 2020Distributional Generalization: A New Kind of Generalizationcited by 41
  15. 2020Unpacking Information Bottlenecks: Unifying Information-Theoretic Objectives in Deep Learningcited by 16
  16. 2021A Practical & Unified Notation for Information-Theoretic Quantities in MLcited by 4
  17. 2021Why Calibration Error is Wrong Given Model Uncertainty: Using Posterior Predictive Checks with Deep Learningcited by 1

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 8; 2024: 2; 2025: 4.

16 papers.

Sort:

By area:

  1. 2024A Survey on Evaluation of Out-of-Distribution Generalizationcited by 20
  2. 2024Toward a Holistic Evaluation of Robustness in CLIP Modelscited by 13
  3. 2023Demystifying Disagreement-on-the-Line in High Dimensionscited by 12
  4. 2025Monitoring Risks in Test-Time Adaptationcited by 10
  5. 2022Task Discovery: Finding the Tasks that Neural Networks Generalize oncited by 9
  6. 2023A Bias-Variance-Covariance Decomposition of Kernel Scores for Generative Modelscited by 8
  7. 2023Inconsistency, Instability, and Generalization Gap of Deep Neural Network Trainingcited by 6
  8. 2023On the Joint Interaction of Models, Data, and Featurescited by 5
  9. 2022Bounding generalization error with input compression: An empirical study with infinite-width networkscited by 4
  10. 2025Error Slice Discovery via Manifold Compactnesscited by 4
  11. 2023Distance Matters For Improving Performance Estimation Under Covariate Shiftcited by 2
  12. 2023Effective Proxy for Human Labeling: Ensemble Disagreement Scores in Large Language Models for Industrial NLPcited by 2
4 more
  1. 2023It's an Alignment, Not a Trade-off: Revisiting Bias and Variance in Deep Modelscited by 1
  2. 2023Beyond Top-Class Agreement: Using Divergences to Forecast Performance under Distribution Shiftcited by 1
  3. 2025Towards Better Generalization via Distributional Input Projection Networkcited by 1
  4. 2025Confidence and Dispersity as Signals: Unsupervised Model Evaluation and Rankingcited by 1