# Impacts of floating-point non-associativity on reproducibility for HPC and deep learning applications arXiv: https://arxiv.org/abs/2408.05148 ยท HTML: https://mlsanity.com/paper/2408.05148/ ## Key facts - Cited by 24 papers in the map (data through 14 Oct 2026). - Builds on 8 of its 31 references that are in the map. - First posted on arXiv 9 Aug 2024. ## Builds on (most cited first, 8 of 8) - [E(3)-Equivariant Graph Neural Networks for Data-Efficient and Accurate Interatomic Potentials](https://mlsanity.com/paper/2101.03164/) (2021) - [DeePMD-kit: A deep learning package for many-body potential energy representation and molecular dynamics](https://mlsanity.com/paper/1712.03641/) (2017) - [Nondeterminism and Instability in Neural Network Optimization](https://mlsanity.com/paper/2103.04514/) (2021) - [Toward Understanding Deep Learning Framework Bugs](https://mlsanity.com/paper/2203.04026/) (2022) - [Silent Bugs in Deep Learning Frameworks: An Empirical Study of Keras and TensorFlow](https://mlsanity.com/paper/2112.13314/) (2021) - [Evaluation of the MACE Force Field Architecture: from Medicinal Chemistry to Materials Science](https://mlsanity.com/paper/2305.14247/) (2023) - [mlf-core: a framework for deterministic machine learning](https://mlsanity.com/paper/2104.07651/) (2021) - [Portability for GPU-accelerated molecular docking applications for cloud and HPC: can portable compiler directives provide performance across all platforms?](https://mlsanity.com/paper/2203.02096/) (2022) ## Built on by (most cited first, 24 of 24) - [Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference](https://mlsanity.com/paper/2506.09501/) (2025) - [Winsor-CAM: Human-Tunable Visual Explanations from Deep Networks via Layer-Wise Winsorization](https://mlsanity.com/paper/2507.10846/) (2025) - [No Soundness in the Real World: On the Challenges of the Verification of Deployed Neural Networks](https://mlsanity.com/paper/2506.01054/) (2025) - [Beyond Reproducibility: Token Probabilities Expose Large Language Model Nondeterminism](https://mlsanity.com/paper/2601.06118/) (2026) - [DASH: Deterministic Attention Scheduling for High-throughput Reproducible LLM Training](https://mlsanity.com/paper/2601.21824/) (2026) - [Bit-Exact AI Inference Verification Without Performance Tradeoffs](https://mlsanity.com/paper/2606.00279/) (2026) - [Dataset-Level Metrics Attenuate Non-Determinism: A Fine-Grained Non-Determinism Evaluation in Diffusion Language Models](https://mlsanity.com/paper/2604.13413/) (2026) - [Kernel Contracts: A Specification Language for ML Kernel Correctness Across Heterogeneous Silicon](https://mlsanity.com/paper/2604.22032/) (2026) - [An Attack to Break Permutation-Based Private Third-Party Inference Schemes for LLMs](https://mlsanity.com/paper/2505.18332/) (2025) - [LLMs can hide text in other text of the same length](https://mlsanity.com/paper/2510.20075/) (2025) - [Floating-Point Neural Network Verification at the Software Level](https://mlsanity.com/paper/2510.23389/) (2025) - [Hawkeye: Reproducing GPU-Level Non-Determinism](https://mlsanity.com/paper/2603.20421/) (2026) - [Introducing Background Temperature to Characterise Hidden Randomness in Large Language Models](https://mlsanity.com/paper/2604.22411/) (2026) - [The Integer Alibi: Localizing Cross-Kernel Divergence in INT8-Quantized LLM Inference](https://mlsanity.com/paper/2608.13756/) (2026) - [Robustness of deep learning classification to adversarial input on GPUs: asynchronous parallel accumulation is a source of vulnerability](https://mlsanity.com/paper/2503.17173/) (2025) - [A solution to generalized learning from small training sets found in infants repeated visual experiences of individual objects](https://mlsanity.com/paper/2510.15060/) (2025) - [A Model-Driven Lossless Compression Algorithm Resistant to Mismatch](https://mlsanity.com/paper/2601.17684/) (2026) - [EigenAI: Deterministic Inference, Verifiable Results](https://mlsanity.com/paper/2602.00182/) (2026) - [Privacy-Preserving Mechanisms Enable Cheap Verifiable Inference of LLMs](https://mlsanity.com/paper/2602.17223/) (2026) - [Numerical Instability and Chaos: Quantifying the Unpredictability of Large Language Models](https://mlsanity.com/paper/2604.13206/) (2026) - [Integrity of peer-to-peer distributed LLM inference under malicious nodes](https://mlsanity.com/paper/2607.19490/) (2026) - [Is INT8 Portable? A Cross-Platform Measurement Study of Quantized Inference on Embedded and Automotive Accelerators](https://mlsanity.com/paper/2609.16085/) (2026) - [Measuring the Checker: Mutation Analysis for GPU-Kernel Benchmark Oracles](https://mlsanity.com/paper/2609.22220/) (2026) - [When Do Models Admit They Are Wrong? Failure Disclosure Is Unstable Under Reinforcement Learning](https://mlsanity.com/paper/2609.33220/) (2026) ## Abstract (arXiv) Run to run variability in parallel programs caused by floating-point non-associativity has been known to significantly affect reproducibility in iterative algorithms, due to accumulating errors. Non-reproducibility can critically affect the efficiency and effectiveness of correctness testing for stochastic programs. Recently, the sensitivity of deep learning training and inference pipelines to floating-point non-associativity has been found to sometimes be extreme. It can prevent certification for commercial applications, accurate assessment of robustness and sensitivity, and bug detection. New approaches in scientific computing applications have coupled deep learning models with high-performance computing, leading to an aggravation of debugging and testing challenges. Here we perform an investigation of the statistical properties of floating-point non-associativity within modern parallel programming models, and analyze performance and productivity impacts of replacing atomic operations with deterministic alternatives on GPUs. We examine the recently-added deterministic options in PyTorch within the context of GPU deployment for deep learning, uncovering and quantifying the impacts of input parameters triggering run to run variability and reporting on the reliability and completeness of the documentation. Finally, we evaluate the strategy of exploiting automatic determinism that could be provided by deterministic hardware, using the Groq accelerator for inference portions of the deep learning pipeline. We demonstrate the benefits that a hardware-based strategy can provide within reproducibility and correctness efforts.