mlsanityData to 7 Oct 2026

Nondeterminism and Instability in Neural Network Optimization

arXiv 2103.04514 ↗First posted 8 Mar 2021cs.LGcited by 47

Not on the map. Why?

Abstract (arXiv)

Nondeterminism in neural network optimization produces uncertainty in performance, making small improvements difficult to discern from run-to-run variability. While uncertainty can be reduced by training multiple model copies, doing so is time-consuming, costly, and harms reproducibility. In this work, we establish an experimental protocol for understanding the effect of optimization nondeterminism on model diversity, allowing us to isolate the effects of a variety of sources of nondeterminism. Surprisingly, we find that all sources of nondeterminism have similar effects on measures of model diversity. To explain this intriguing fact, we identify the instability of model training, taken as an end-to-end procedure, as the key determinant. We show that even one-bit changes in initial parameters result in models converging to vastly different values. Last, we propose two approaches for reducing the effects of instability on run-to-run variability.

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). Builds on 24 of its 40 references that are in the map. First posted on arXiv 8 Mar 2021.

Builds on

24 of 40 references.

References · 24

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  6. 2014Going Deeper with Convolutionscited by 4,231
  7. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  8. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  9. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  10. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,132
  11. 2017Deep Reinforcement Learning that Matterscited by 727
  12. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
12 more
  1. 2019Deep Ensembles: A Loss Landscape Perspectivecited by 438
  2. 2020BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learningcited by 368
  3. 2017Snapshot Ensembles: Train 1, get M for freecited by 349
  4. 2017Stable Architectures for Deep Neural Networkscited by 340
  5. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 267
  6. 2014cuDNN: Efficient Primitives for Deep Learningcited by 240
  7. 2017Regularizing and Optimizing LSTM Language Modelscited by 205
  8. 2020The Early Phase of Neural Network Trainingcited by 114
  9. 2017Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Controlcited by 92
  10. 2016Quasi-Recurrent Neural Networkscited by 87
  11. 2017Critical Learning Periods in Deep Neural Networkscited by 78
  12. 2019On Model Stability as a Function of Random Seedcited by 31

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 9; 2023: 13; 2024: 3; 2025: 11; 2026: 9.

47 papers.

Sort:

  1. 2022Metrics reloaded: Recommendations for image analysis validationcited by 190
  2. 2022Training Data Influence Analysis and Estimation: A Surveycited by 134
  3. 2023Similarity of Neural Network Models: A Survey of Functional and Representational Measurescited by 128
  4. 2022Data Banzhaf: A Robust Data Valuation Framework for Machine Learningcited by 126
  5. 2023Elo Uncovered: Robustness and Best Practices in Language Model Evaluationcited by 58
  6. 2021Randomness In Neural Network Training: Characterizing The Impact of Toolingcited by 57
  7. 2025Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inferencecited by 47
  8. 2022On the Factory Floor: ML Engineering for Industrial-Scale Ads Recommendation Modelscited by 34
  9. 2024Impacts of floating-point non-associativity on reproducibility for HPC and deep learning applicationscited by 24
  10. 2025Pre-training under infinite computecited by 24
  11. 2022Reproducibility in Optimization: Theoretical Framework and Limitscited by 21
  12. 2023On the Variance of Neural Network Training with respect to Test Sets and Distributionscited by 20
35 more
  1. 2023The Co-12 Recipe for Evaluating Interpretable Part-Prototype Image Classifierscited by 15
  2. 2023A Survey on Stability of Learning with Limited Labelled Data and its Sensitivity to the Effects of Randomnesscited by 14
  3. 2023FAIR-Ensemble: When Fairness Naturally Emerges From Deep Ensemblingcited by 11
  4. 2023Measuring the Instability of Fine-Tuningcited by 7
  5. 2022On the Prediction Instability of Graph Neural Networkscited by 6
  6. 2023An algorithmic framework for the optimization of deep neural networks architectures and hyperparameterscited by 6
  7. 2022Instability in clinical risk stratification models using deep learningcited by 3
  8. 2022Real World Large Scale Recommendation Systems Reproducibility and Smooth Activationscited by 2
  9. 2022Statistical Physics of Deep Neural Networks: Initialization toward Optimal Channelscited by 2
  10. 2023Transferring Learning Trajectories of Neural Networkscited by 2
  11. 2023On The Fairness Impacts of Hardware Selection in Machine Learningcited by 2
  12. 2025RepDL: Bit-level Reproducible Deep Learning Training and Inferencecited by 2
  13. 2023Robust Nonparametric Hypothesis Testing to Understand Variability in Training Neural Networkscited by 1
  14. 2025Robustness of deep learning classification to adversarial input on GPUs: asynchronous parallel accumulation is a source of vulnerabilitycited by 1
  15. 2025Why are LLMs' abilities emergent?cited by 1
  16. 2025Bit-Accurate Modeling of GPU Matrix Multiply-Accumulate Units: Demystifying Numerical Discrepancy and Accuracycited by 1
  17. 2026PIDSMaker: Building and Evaluating Provenance-based Intrusion Detection Systemscited by 1
  18. 2026q0: Primitives for Hyper-Epoch Pretrainingcited by 1
  19. 2026Robust Ambiguity Detection (RAD) From Model- and Feature-Space Consistencycited by 1
  20. 2021Dropout Prediction Uncertainty Estimation Using Neuron Activation Strengthcited by 0
  21. 2023Examining the Effect of Pre-training on Time Series Classificationcited by 0
  22. 2023Distilling Influences to Mitigate Prediction Churn in Graph Neural Networkscited by 0
  23. 2024Measuring training variability from stochastic optimization using robust nonparametric testingcited by 0
  24. 2024The Impact of Environment Configurations on the Stability of AI-Enabled Systemscited by 0
  25. 2025Improving Out-of-Distribution Detection with Markov Logic Networkscited by 0
  26. 2025Fractal basin geometry and the limits to predictability and reproducibility in deep learningcited by 0
  27. 2025A solution to generalized learning from small training sets found in infants repeated visual experiences of individual objectscited by 0
  28. 2025Revisiting the Relation Between Robustness and Universalitycited by 0
  29. 2025Embedding Explainable AI in NHS Clinical Safety: The Explainability-Enabled Clinical Safety Framework (ECSF)cited by 0
  30. 2026Learning to be Reproducible: Custom Loss Design for Robust Neural Networkscited by 0
  31. 2026Dynamic Scaled Gradient Descent for Stable Fine-Tuning for Classificationscited by 0
  32. 2026Quantifying the Impact of Lossy Compression on Neural Generative Surrogate Modelingcited by 0
  33. 2026GRAIN: Group Aggregation via Min-Norm Objectivecited by 0
  34. 2026Rating the Pitch, Not the Product: User Evaluations of LLMs Reflect Expectations More Than Performancecited by 0
  35. 2026Same Probe, Different Numbers: Are Activation Probes Robust to Inference-Time Numerical Non-Determinism?cited by 0