# Bit-Accurate Modeling of GPU Matrix Multiply-Accumulate Units: Demystifying Numerical Discrepancy and Accuracy

arXiv: https://arxiv.org/abs/2511.10909 · HTML: https://mlsanity.com/paper/2511.10909/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 32 references that are in the map.
- First posted on arXiv 14 Nov 2025.

## Builds on (most cited first, 15 of 15)

- [In-Datacenter Performance Analysis of a Tensor Processing Unit](https://mlsanity.com/paper/1704.04760/) (2017)
- [FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision](https://mlsanity.com/paper/2407.08608/) (2024)
- [SageAttention2: Efficient Attention with Thorough Outlier Smoothing and Per-thread INT4 Quantization](https://mlsanity.com/paper/2411.10958/) (2024)
- [NVIDIA Tensor Core Programmability, Performance & Precision](https://mlsanity.com/paper/1803.04014/) (2018)
- [Randomness In Neural Network Training: Characterizing The Impact of Tooling](https://mlsanity.com/paper/2106.11872/) (2021)
- [Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference](https://mlsanity.com/paper/2506.09501/) (2025)
- [Nondeterminism and Instability in Neural Network Optimization](https://mlsanity.com/paper/2103.04514/) (2021)
- [Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures](https://mlsanity.com/paper/2505.09343/) (2025)
- [Towards Training Reproducible Deep Learning Models](https://mlsanity.com/paper/2202.02326/) (2022)
- [Modeling Deep Learning Accelerator Enabled GPUs](https://mlsanity.com/paper/1811.08309/) (2018)
- [Dissecting Tensor Cores via Microbenchmarks: Latency, Throughput and Numeric Behaviors](https://mlsanity.com/paper/2206.02874/) (2022)
- [Testing GPU Numerics: Finding Numerical Differences Between NVIDIA and AMD GPUs](https://mlsanity.com/paper/2410.09172/) (2024)
- [An SMT Formalization of Mixed-Precision Matrix Multiplication: Modeling Three Generations of Tensor Cores](https://mlsanity.com/paper/2502.15999/) (2025)
- [Revealing Floating-Point Accumulation Orders in Software/Hardware Implementations](https://mlsanity.com/paper/2411.00442/) (2024)
- [FTTN: Feature-Targeted Testing for Numerical Properties of NVIDIA & AMD Matrix Accelerators](https://mlsanity.com/paper/2403.00232/) (2024)

## Built on by (most cited first, 1 of 1)

- [Bit-Exact AI Inference Verification Without Performance Tradeoffs](https://mlsanity.com/paper/2606.00279/) (2026)

## Abstract (arXiv)

Modern AI accelerators rely on matrix multiply-accumulate units (MMAUs), such as NVIDIA Tensor Cores and AMD Matrix Cores, to accelerate deep neural network workloads. MMAUs expose only instruction-level or API-level interfaces of matrix multiply-accumulate (MMA) operations, while leaving internal floating-point arithmetic behavior undocumented. Consequently, MMAUs across vendors and architectural generations often produce numerical discrepancies for identical inputs, and sometimes exhibit reduced numerical accuracy that can cause training instability. Diagnosing and understanding the root causes of these effects is challenging without white-box models of their arithmetic behavior. This paper proposes closed-loop feature probing (CLFP), a generic and systematic framework for constructing bit-accurate arithmetic behavior models of MMA operations. Based on this framework, we analyze all MMA instructions on ten GPU architectures spanning NVIDIA Volta through RTX Blackwell and AMD CDNA1 through CDNA3, and derive the first bit-accurate arithmetic models for these MMAUs. Our models explain previously observed cross-platform numerical discrepancies and accuracy issues, enable white-box numerical error analysis, reveal four types of precision bottlenecks and one type of numerical asymmetry, and inform software workarounds as well as design suggestions for future MMAUs. This work is open-source at https://github.com/microsoft/MMA-Sim