# A Note on "Assessing Generalization of SGD via Disagreement"

arXiv: https://arxiv.org/abs/2202.01851 · HTML: https://mlsanity.com/paper/2202.01851/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 46 references that are in the map.
- First posted on arXiv 3 Feb 2022.

## Builds on (most cited first, 29 of 29)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks](https://mlsanity.com/paper/1807.03888/) (2018)
- [Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks](https://mlsanity.com/paper/1706.02690/) (2017)
- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [Deeper, Broader and Artier Domain Generalization](https://mlsanity.com/paper/1710.03077/) (2017)
- [Deep Bayesian Active Learning with Image Data](https://mlsanity.com/paper/1703.02910/) (2017)
- [Bayesian Deep Learning and a Probabilistic Perspective of Generalization](https://mlsanity.com/paper/2002.08791/) (2020)
- [Measuring Calibration in Deep Learning](https://mlsanity.com/paper/1904.01685/) (2019)
- [BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learning](https://mlsanity.com/paper/1906.08158/) (2019)
- [Bag of Tricks for Image Classification with Convolutional Neural Networks](https://mlsanity.com/paper/1812.01187/) (2018)
- [Understanding Measures of Uncertainty for Adversarial Example Detection](https://mlsanity.com/paper/1803.08533/) (2018)
- [Leveraging Unlabeled Data to Predict Out-of-Distribution Performance](https://mlsanity.com/paper/2201.04234/) (2022)
- [Assessing Generalization of SGD via Disagreement](https://mlsanity.com/paper/2106.13799/) (2021)
- [Detecting Errors and Estimating Accuracy on Unlabeled Data with Self-training Ensembles](https://mlsanity.com/paper/2106.15728/) (2021)
- [DOCTOR: A Simple Method for Detecting Misclassification Errors](https://mlsanity.com/paper/2106.02395/) (2021)
- [Distributional Generalization: A New Kind of Generalization](https://mlsanity.com/paper/2009.08092/) (2020)
- [Unpacking Information Bottlenecks: Unifying Information-Theoretic Objectives in Deep Learning](https://mlsanity.com/paper/2003.12537/) (2020)
- [A Practical & Unified Notation for Information-Theoretic Quantities in ML](https://mlsanity.com/paper/2106.12062/) (2021)
- [Why Calibration Error is Wrong Given Model Uncertainty: Using Posterior Predictive Checks with Deep Learning](https://mlsanity.com/paper/2112.01477/) (2021)

## Built on by (most cited first, 16 of 16)

- [A Survey on Evaluation of Out-of-Distribution Generalization](https://mlsanity.com/paper/2403.01874/) (2024)
- [Toward a Holistic Evaluation of Robustness in CLIP Models](https://mlsanity.com/paper/2410.01534/) (2024)
- [Demystifying Disagreement-on-the-Line in High Dimensions](https://mlsanity.com/paper/2301.13371/) (2023)
- [Monitoring Risks in Test-Time Adaptation](https://mlsanity.com/paper/2507.08721/) (2025)
- [Task Discovery: Finding the Tasks that Neural Networks Generalize on](https://mlsanity.com/paper/2212.00261/) (2022)
- [A Bias-Variance-Covariance Decomposition of Kernel Scores for Generative Models](https://mlsanity.com/paper/2310.05833/) (2023)
- [Inconsistency, Instability, and Generalization Gap of Deep Neural Network Training](https://mlsanity.com/paper/2306.00169/) (2023)
- [On the Joint Interaction of Models, Data, and Features](https://mlsanity.com/paper/2306.04793/) (2023)
- [Bounding generalization error with input compression: An empirical study with infinite-width networks](https://mlsanity.com/paper/2207.09408/) (2022)
- [Error Slice Discovery via Manifold Compactness](https://mlsanity.com/paper/2501.19032/) (2025)
- [Distance Matters For Improving Performance Estimation Under Covariate Shift](https://mlsanity.com/paper/2308.07223/) (2023)
- [Effective Proxy for Human Labeling: Ensemble Disagreement Scores in Large Language Models for Industrial NLP](https://mlsanity.com/paper/2309.05619/) (2023)
- [It's an Alignment, Not a Trade-off: Revisiting Bias and Variance in Deep Models](https://mlsanity.com/paper/2310.09250/) (2023)
- [Beyond Top-Class Agreement: Using Divergences to Forecast Performance under Distribution Shift](https://mlsanity.com/paper/2312.08033/) (2023)
- [Towards Better Generalization via Distributional Input Projection Network](https://mlsanity.com/paper/2506.04690/) (2025)
- [Confidence and Dispersity as Signals: Unsupervised Model Evaluation and Ranking](https://mlsanity.com/paper/2510.02956/) (2025)

## Abstract (arXiv)

Several recent works find empirically that the average test error of deep neural networks can be estimated via the prediction disagreement of models, which does not require labels. In particular, Jiang et al. (2022) show for the disagreement between two separately trained networks that this `Generalization Disagreement Equality' follows from the well-calibrated nature of deep ensembles under the notion of a proposed `class-aggregated calibration.' In this reproduction, we show that the suggested theory might be impractical because a deep ensemble's calibration can deteriorate as prediction disagreement increases, which is precisely when the coupling of test error and disagreement is of interest, while labels are needed to estimate the calibration on new datasets. Further, we simplify the theoretical statements and proofs, showing them to be straightforward within a probabilistic context, unlike the original hypothesis space view employed by Jiang et al. (2022).