# Non-Determinism in TensorFlow ResNets

arXiv: https://arxiv.org/abs/2001.11396 · HTML: https://mlsanity.com/paper/2001.11396/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Jan 2020.

## Built on by (most cited first, 11 of 11)

- [On the Reproducibility of Neural Network Predictions](https://mlsanity.com/paper/2102.03349/) (2021)
- [Generative Models as a Complex Systems Science: How can we make sense of large language model behavior?](https://mlsanity.com/paper/2308.00189/) (2023)
- [DORIS-MAE: Scientific Document Retrieval using Multi-level Aspect-based Queries](https://mlsanity.com/paper/2310.04678/) (2023)
- [Reproducibility of Machine Learning: Terminology, Recommendations and Open Issues](https://mlsanity.com/paper/2302.12691/) (2023)
- [Engineering the Neural Automatic Passenger Counter](https://mlsanity.com/paper/2203.01156/) (2022)
- [FEEL: A Framework for Evaluating Emotional Support Capability with Large Language Models](https://mlsanity.com/paper/2403.15699/) (2024)
- [Evaluating Generic Auto-ML Tools for Computational Pathology](https://mlsanity.com/paper/2112.03622/) (2021)
- [Impact of GPU uncertainty on the training of predictive deep neural networks](https://mlsanity.com/paper/2109.01451/) (2021)
- [Incorporating Crowdsourced Annotator Distributions into Ensemble Modeling to Improve Classification Trustworthiness for Ancient Greek Papyri](https://mlsanity.com/paper/2210.16380/) (2022)
- [A Model-Driven Lossless Compression Algorithm Resistant to Mismatch](https://mlsanity.com/paper/2601.17684/) (2026)
- [Omega-S: A Functional Resilience Index for LLM Fine-Tuning](https://mlsanity.com/paper/2608.03887/) (2026)

## Abstract (arXiv)

We show that the stochasticity in training ResNets for image classification on GPUs in TensorFlow is dominated by the non-determinism from GPUs, rather than by the initialisation of the weights and biases of the network or by the sequence of minibatches given. The standard deviation of test set accuracy is 0.02 with fixed seeds, compared to 0.027 with different seeds---nearly 74% of the standard deviation of a ResNet model is non-deterministic. For test set loss the ratio of standard deviations is more than 80%. These results call for more robust evaluation strategies of deep learning models, as a significant amount of the variation in results across runs can arise simply from GPU randomness.