# Hawkeye: Reproducing GPU-Level Non-Determinism

arXiv: https://arxiv.org/abs/2603.20421 · HTML: https://mlsanity.com/paper/2603.20421/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 44 references that are in the map.
- First posted on arXiv 20 Mar 2026.

## Builds on (most cited first, 14 of 14)

- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [ZeRO: Memory Optimizations Toward Training Trillion Parameter Models](https://mlsanity.com/paper/1910.02054/) (2019)
- [zkLLM: Zero Knowledge Proofs for Large Language Models](https://mlsanity.com/paper/2404.16109/) (2024)
- [Impacts of floating-point non-associativity on reproducibility for HPC and deep learning applications](https://mlsanity.com/paper/2408.05148/) (2024)
- [A Survey of Zero-Knowledge Proof Based Verifiable Machine Learning](https://mlsanity.com/paper/2502.18535/) (2025)
- [TOPLOC: A Locality Sensitive Hashing Scheme for Trustless Verifiable Inference](https://mlsanity.com/paper/2501.16007/) (2025)
- [Verde: Verification via Refereed Delegation for Machine Learning Programs](https://mlsanity.com/paper/2502.19405/) (2025)
- [Optimistic Verifiable Training by Controlling Hardware Nondeterminism](https://mlsanity.com/paper/2403.09603/) (2024)
- [Artemis: Efficient Commit-and-Prove SNARKs for zkML](https://mlsanity.com/paper/2409.12055/) (2024)
- [Trust the Process: Zero-Knowledge Machine Learning to Enhance Trust in Generative AI Interactions](https://mlsanity.com/paper/2402.06414/) (2024)
- [mlf-core: a framework for deterministic machine learning](https://mlsanity.com/paper/2104.07651/) (2021)
- [opp/ai: Optimistic Privacy-Preserving AI on Blockchain](https://mlsanity.com/paper/2402.15006/) (2024)

## Built on by (most cited first, 2 of 2)

- [The Integer Alibi: Localizing Cross-Kernel Divergence in INT8-Quantized LLM Inference](https://mlsanity.com/paper/2608.13756/) (2026)
- [Accelerating the Mitigation of LLM Inference Nondeterminism Across GPU Architectures](https://mlsanity.com/paper/2609.25624/) (2026)

## Abstract (arXiv)

We present Hawkeye, a system for analyzing and reproducing GPU-level arithmetic operations. Using our framework, anyone can re-execute on a CPU the exact matrix multiplication operations underlying a machine learning model training or inference workflow that was executed on an NVIDIA GPU, without any precision loss. This is in stark contrast to prior approaches to verifiable machine learning, which either introduce significant computation overhead to the original model owner, or suffer from non-robustness and quality degradation. The main technical contribution of Hawkeye is a systematic sequence of carefully crafted tests that study rounding direction, subnormal number handling, and order of (non-associative) accumulation during matrix multiplication on NVIDIA's Tensor Cores. We test and evaluate our framework on multiple NVIDIA GPU architectures ( Ampere, Hopper, and Lovelace) and precision types (FP16, BFP16, FP8). In all test cases, Hawkeye enables perfect reproduction of matrix multiplication on a CPU, paving the way for efficient and trustworthy third-party auditing of ML model training and inference. We provide source code for Hawkeye at https://github.com/badasherez/gpu-simulator.