Robust Nonparametric Hypothesis Testing to Understand Variability in Training Neural Networks
Not on the map. Why?
Abstract (arXiv)
Training a deep neural network (DNN) often involves stochastic optimization, which means each run will produce a different model. Several works suggest this variability is negligible when models have the same performance, which in the case of classification is test accuracy. However, models with similar test accuracy may not be computing the same function. We propose a new measure of closeness between classification models based on the output of the network before thresholding. Our measure is based on a robust hypothesis-testing framework and can be adapted to other quantities derived from trained models.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 10 of its 20 references that are in the map. First posted on arXiv 1 Oct 2023.
Builds on
10 of 20 references.
References · 10
- 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
- 2019Deep Ensembles: A Loss Landscape Perspectivecited by 438
- 2021Accounting for Variance in Machine Learning Benchmarkscited by 195
- 2015Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networkscited by 138
- 2018High-Quality Prediction Intervals for Deep Learning: A Distribution-Free, Ensembled Approachcited by 117
- 2022Can Neural Nets Learn the Same Model Twice? Investigating Reproducibility and Double Descent from the Decision Boundary Perspectivecited by 57
- 2021Nondeterminism and Instability in Neural Network Optimizationcited by 47
- 2022Sources of Irreproducibility in Machine Learning: A Reviewcited by 28
- 2023On the Variance of Neural Network Training with respect to Test Sets and Distributionscited by 20
- 2019On approximate validation of models: A Kolmogorov-Smirnov based approachcited by 3
Built on by
Citing papers per year: 2024: 1.
1 paper.