Leveraging Gradients for Unsupervised Accuracy Estimation under Distribution Shift
Not on the map. Why?
Abstract (arXiv)
Estimating the test performance of a model, possibly under distribution shift, without having access to the ground-truth labels is a challenging, yet very important problem for the safe deployment of machine learning algorithms in the wild. Existing works mostly rely on information from either the outputs or the extracted features of neural networks to estimate a score that correlates with the ground-truth test accuracy. In this paper, we investigate -- both empirically and theoretically -- how the information provided by the gradients can be predictive of the ground-truth test accuracy even under distribution shifts. More specifically, we use the norm of classification-layer gradients, backpropagated from the cross-entropy loss after only one gradient step over test data. Our intuition is that these gradients should be of higher magnitude when the model generalizes poorly. We provide the theoretical insights behind our approach and the key ingredients that ensure its empirical success. Extensive experiments conducted with various architectures on diverse distribution shifts demonstrate that our method significantly outperforms current state-of-the-art approaches. The code is available at https://github.com/Renchunzi-Xie/GdScore
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 50 of its 64 references that are in the map. First posted on arXiv 17 Jan 2024.
Builds on
50 of 64 references.
References · 50
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
- 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
- 2017On Calibration of Modern Neural Networkscited by 4,860
- 2016Wide Residual Networkscited by 3,057
- 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
- 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
- 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,080
- 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
- 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,366
- 2017Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networkscited by 1,325
- 2020Energy-based Out-of-distribution Detectioncited by 1,296
38 more
- 2017Deep Hashing Network for Unsupervised Domain Adaptationcited by 1,181
- 2019When Does Label Smoothing Help?cited by 985
- 2018Deep Anomaly Detection with Outlier Exposurecited by 939
- 2021Generalized Out-of-Distribution Detection: A Surveycited by 796
- 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 671
- 2017Exploring Generalization in Deep Learningcited by 595
- 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 547
- 2021Revisiting the Calibration of Modern Neural Networkscited by 387
- 2017Measuring Catastrophic Forgetting in Neural Networkscited by 359
- 2021On the Importance of Gradients for Detecting Distributional Shifts in the Wildcited by 338
- 2021Gradient Matching for Domain Generalizationcited by 258
- 2020BREEDS: Benchmarks for Subpopulation Shiftcited by 155
- 2022Leveraging Unlabeled Data to Predict Out-of-Distribution Performancecited by 141
- 2022Penalizing Gradient Norm for Efficiently Improving Generalization in Deep Learningcited by 122
- 2019Information-Theoretic Generalization Bounds for SGLD via Data-Dependent Estimatescited by 118
- 2021Assessing Generalization of SGD via Disagreementcited by 116
- 2021Predicting with Confidence on Unseen Distributionscited by 116
- 2018Predicting the Generalization Gap in Deep Networks with Margin Distributionscited by 111
- 2020Predicting Neural Network Accuracy from Weightscited by 107
- 2020Are Labels Always Necessary for Classifier Accuracy Evaluation?cited by 104
- 2022Self-Training: A Surveycited by 85
- 2019Provable Guarantees for Gradient-Based Meta-Learningcited by 67
- 2021What Does Rotation Prediction Tell Us about Classifier Accuracy under Varying Testing Environments?cited by 63
- 2021Domain Generalization via Gradient Surgerycited by 60
- 2019On Generalization Error Bounds of Noisy Gradient Methods for Non-Convex Learningcited by 57
- 2020Computing the Testing Error without a Testing Setcited by 54
- 2021Detecting Errors and Estimating Accuracy on Unlabeled Data with Self-training Ensemblescited by 54
- 2022Predicting Out-of-Distribution Error with the Projection Normcited by 50
- 2019Learning-to-Learn Stochastic Gradient Descent with Biased Regularizationcited by 49
- 2019Heavy-Tailed Universality Predicts Trends in Test Accuracies for Very Large Pre-Trained Deep Neural Networkscited by 43
- 2020Coherent Gradients: An Approach to Understanding Generalization in Gradient Descent-based Optimizationcited by 41
- 2017A PAC-Bayesian Analysis of Randomized Learning with Application to Stochastic Gradient Descentcited by 29
- 2017Estimating Accuracy from Unlabeled Data: A Probabilistic Logic Approachcited by 27
- 2023Confidence and Dispersity Speak: Characterising Prediction Matrix for Unsupervised Accuracy Estimationcited by 22
- 2018PAC-Bayes bounds for stable algorithms with instance-dependent priorscited by 21
- 2023Characterizing Out-of-Distribution Error via Optimal Transportcited by 20
- 2019Towards Task and Architecture-Independent Generalization Gap Predictorscited by 18
- 2023On the Importance of Feature Separability in Predicting Out-Of-Distribution Errorcited by 15
Built on by
Citing papers per year: 2024: 1; 2025: 0; 2026: 2.
3 papers.