# Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference

arXiv: https://arxiv.org/abs/2506.09501 · HTML: https://mlsanity.com/paper/2506.09501/

## Key facts

- Cited by 53 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 39 references that are in the map.
- First posted on arXiv 11 Jun 2025.

## Builds on (most cited first, 23 of 23)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models](https://mlsanity.com/paper/2503.16419/) (2025)
- [An Empirical Study of the Non-determinism of ChatGPT in Code Generation](https://mlsanity.com/paper/2308.02828/) (2023)
- [A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibility](https://mlsanity.com/paper/2504.07086/) (2025)
- [Randomness In Neural Network Training: Characterizing The Impact of Tooling](https://mlsanity.com/paper/2106.11872/) (2021)
- [Nondeterminism and Instability in Neural Network Optimization](https://mlsanity.com/paper/2103.04514/) (2021)
- [Impacts of floating-point non-associativity on reproducibility for HPC and deep learning applications](https://mlsanity.com/paper/2408.05148/) (2024)
- [How Numerical Precision Affects Arithmetical Reasoning Capabilities of LLMs](https://mlsanity.com/paper/2410.13857/) (2024)
- [Assessing Consistency and Reproducibility in the Outputs of Large Language Models: Evidence Across Diverse Finance and Accounting Tasks](https://mlsanity.com/paper/2503.16974/) (2025)
- [Optimizing Large Language Model Hyperparameters for Code Generation](https://mlsanity.com/paper/2408.10577/) (2024)
- [Studying How Configurations Impact Code Generation in LLMs: the Case of ChatGPT](https://mlsanity.com/paper/2502.17450/) (2025)
- [Measuring Determinism in Large Language Models for Software Code Review](https://mlsanity.com/paper/2502.20747/) (2025)
- [The Science of Evaluating Foundation Models](https://mlsanity.com/paper/2502.09670/) (2025)
- [Benchmarking Large Language Model Volatility](https://mlsanity.com/paper/2311.15180/) (2023)

## Built on by (most cited first, 30 of 53)

- [On Randomness in Agentic Evals](https://mlsanity.com/paper/2602.07150/) (2026)
- [Deterministic Inference across Tensor Parallel Sizes That Eliminates Training-Inference Mismatch](https://mlsanity.com/paper/2511.17826/) (2025)
- [Hidden Measurement Error in LLM Pipelines Distorts Annotation, Evaluation, and Benchmarking](https://mlsanity.com/paper/2604.11581/) (2026)
- [The Silent Hyperparameter: Quantifying the Impact of Inference Backends on LLM Reproducibility](https://mlsanity.com/paper/2605.19537/) (2026)
- [When LLMs get significantly worse: A statistical approach to detect model degradations](https://mlsanity.com/paper/2602.10144/) (2026)
- [The Illusion of Equivalence: Systematic FP16 Divergence in KV-Cached Autoregressive Inference](https://mlsanity.com/paper/2604.15409/) (2026)
- [Dataset-Level Metrics Attenuate Non-Determinism: A Fine-Grained Non-Determinism Evaluation in Diffusion Language Models](https://mlsanity.com/paper/2604.13413/) (2026)
- [Kernel Contracts: A Specification Language for ML Kernel Correctness Across Heterogeneous Silicon](https://mlsanity.com/paper/2604.22032/) (2026)
- [MarginGate: Sparse Margin-Triggered Verification for Batch-Invariant LLM Inference](https://mlsanity.com/paper/2605.30218/) (2026)
- [Hardware-Triggered Backdoors](https://mlsanity.com/paper/2601.21902/) (2026)
- [SEAR: Simple and Efficient Adaptation of Visual Geometric Transformers for Unpaired RGB+Thermal 3D Reconstruction](https://mlsanity.com/paper/2603.18774/) (2026)
- [Paraphrase Brittleness in Production Retrieval-Augmented Commercial Recommendation: Reproducibility Below the Rerun-Stability Baseline](https://mlsanity.com/paper/2605.27440/) (2026)
- [FloatDoor: Platform-Triggered Backdoors in LLMs](https://mlsanity.com/paper/2606.19535/) (2026)
- [Demystifying Numerical Instability in LLM Inference: Achieving Reproducible Inference for Mission-Critical Tasks with HEAL](https://mlsanity.com/paper/2606.21023/) (2026)
- [AgentTrails: Towards Trust and Reuse for Agentic Tasks](https://mlsanity.com/paper/2607.18816/) (2026)
- [Kalypso: Relational LLM Serving](https://mlsanity.com/paper/2607.23815/) (2026)
- [Do All LLMs Know When They're Being Harmful? A Reproducibility Study of Latent-Space Safety Probes Across Model Families](https://mlsanity.com/paper/2608.08029/) (2026)
- [The Integer Alibi: Localizing Cross-Kernel Divergence in INT8-Quantized LLM Inference](https://mlsanity.com/paper/2608.13756/) (2026)
- [Bit-Accurate Modeling of GPU Matrix Multiply-Accumulate Units: Demystifying Numerical Discrepancy and Accuracy](https://mlsanity.com/paper/2511.10909/) (2025)
- [Fingerprinting Inference Systems of Large Language Models](https://mlsanity.com/paper/2605.29979/) (2026)
- [Re-feeding Is Not Replaying: Measuring Replay Noise in Counterfactual Token-Credit Estimation](https://mlsanity.com/paper/2606.15621/) (2026)
- [M-GATE: Multilingual Grammar, Accuracy in Translation, and Efficiency Benchmark for Large Language Models](https://mlsanity.com/paper/2608.03803/) (2026)
- [Routing Is Least Learnable Where It Is Most Valuable: Bounds on Representation Routing for Web Agents](https://mlsanity.com/paper/2608.06171/) (2026)
- [Benchmarking LLM Serving Systems for Agentic AI Workloads with XPerf](https://mlsanity.com/paper/2608.20370/) (2026)
- [Deterministic LLM Inference Across GPU Kernels: Power-of-Two INT8 Quantization Scales and the Limits of Tolerance-Based Conformance](https://mlsanity.com/paper/2609.00363/) (2026)
- [Accelerating the Mitigation of LLM Inference Nondeterminism Across GPU Architectures](https://mlsanity.com/paper/2609.25624/) (2026)
- [Dating the Model: Hidden Dates in System Prompts Affect LLM Evaluation](https://mlsanity.com/paper/2609.36931/) (2026)
- [A solution to generalized learning from small training sets found in infants repeated visual experiences of individual objects](https://mlsanity.com/paper/2510.15060/) (2025)
- [Information Theoretic Perspective on Representation Learning](https://mlsanity.com/paper/2601.11334/) (2026)
- [A Note on Non-Composability of Layerwise Approximate Verification for Neural Inference](https://mlsanity.com/paper/2602.15756/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) are now integral across various domains and have demonstrated impressive performance. Progress, however, rests on the premise that benchmark scores are both accurate and reproducible. We demonstrate that the reproducibility of LLM performance is fragile: changing system configuration, such as evaluation batch size, GPU count, and GPU version, can introduce significant differences in the generated responses. This issue is especially pronounced in reasoning models, where minor rounding differences in early tokens can cascade into divergent chains of thought, ultimately affecting accuracy. For instance, under bfloat16 precision with greedy decoding, a reasoning model like DeepSeek-R1-Distill-Qwen-7B can exhibit up to 9% variation in accuracy and 9,000 tokens difference in response length due to differences in GPU count, type, and evaluation batch size. We trace the root cause of this variability to the non-associative nature of floating-point arithmetic under limited numerical precision. This work presents the first systematic investigation into how numerical precision affects reproducibility in LLM inference. Through carefully controlled experiments across various hardware, software, and precision settings, we quantify when and how model outputs diverge. Our analysis reveals that floating-point precision - while critical for reproducibility - is often neglected in evaluation practices. Inspired by this, we develop a lightweight inference pipeline, dubbed LayerCast, that stores weights in 16-bit precision but performs all computations in FP32, balancing memory efficiency with numerical stability. Code is available at https://github.com/nanomaoli/llm_reproducibility.