# Mixed precision accumulation for neural network inference guided by componentwise forward error analysis

arXiv: https://arxiv.org/abs/2503.15568 · HTML: https://mlsanity.com/paper/2503.15568/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 39 references that are in the map.
- First posted on arXiv 19 Mar 2025.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [A White Paper on Neural Network Quantization](https://mlsanity.com/paper/2106.08295/) (2021)
- [HAQ: Hardware-Aware Automated Quantization with Mixed Precision](https://mlsanity.com/paper/1811.08886/) (2018)
- [HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precision](https://mlsanity.com/paper/1905.03696/) (2019)
- [Deep Learning with Limited Numerical Precision](https://mlsanity.com/paper/1502.02551/) (2015)
- [HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networks](https://mlsanity.com/paper/1911.03852/) (2019)
- [Training Deep Neural Networks with 8-bit Floating Point Numbers](https://mlsanity.com/paper/1812.08011/) (2018)
- [Fixed Point Quantization of Deep Convolutional Networks](https://mlsanity.com/paper/1511.06393/) (2015)
- [BSQ: Exploring Bit-Level Sparsity for Mixed-Precision Neural Network Quantization](https://mlsanity.com/paper/2102.10462/) (2021)
- [Accumulation Bit-Width Scaling For Ultra-Low Precision Training Of Deep Networks](https://mlsanity.com/paper/1901.06588/) (2019)
- [Quantization of Deep Neural Networks for Accumulator-constrained Processors](https://mlsanity.com/paper/2004.11783/) (2020)
- [Numerical stability analysis of large language models](https://mlsanity.com/paper/2503.10251/) (2025)
- [Accumulator-Aware Post-Training Quantization for Large Language Models](https://mlsanity.com/paper/2409.17092/) (2024)
- [Quantized Neural Networks for Low-Precision Accumulation with Guaranteed Overflow Avoidance](https://mlsanity.com/paper/2301.13376/) (2023)

## Built on by (most cited first, 6 of 6)

- [Fuzzy PyTorch: Rapid Numerical Variability Evaluation for Deep Learning Models](https://mlsanity.com/paper/2605.25991/) (2026)
- [LAMP: Look-Ahead Mixed-Precision Inference of Large Language Models](https://mlsanity.com/paper/2601.21623/) (2026)
- [Operator-Aware Mixed-Precision Tolerance Calibration for Tensor Kernels](https://mlsanity.com/paper/2607.16228/) (2026)
- [Lipschitz-Based Robustness Certification Under Floating-Point Execution](https://mlsanity.com/paper/2603.13334/) (2026)
- [Automated Numerical Stability Analysis of Deep Learning Operators](https://mlsanity.com/paper/2607.25494/) (2026)
- [LampAttention: Look-Ahead Mixed-Precision FlashAttention for Dedicated Accelerators](https://mlsanity.com/paper/2609.39361/) (2026)

## Abstract (arXiv)

This work proposes a mathematically founded mixed precision accumulation strategy for the inference of neural networks. Our strategy is based on a new componentwise forward error analysis that explains the propagation of errors in the forward pass of neural networks. Specifically, our analysis shows that the error in each component of the output of a linear layer is proportional to the condition number of the inner product between the weights and the input, multiplied by the condition number of the activation function. These condition numbers can vary widely from one component to the other, thus creating a significant opportunity to introduce mixed precision: each component should be accumulated in a precision inversely proportional to the product of these condition numbers. We propose a numerical algorithm that exploits this observation: it first computes all components in low precision, uses this output to estimate the condition numbers, and recomputes in higher precision only the components associated with large condition numbers. We test our algorithm on various networks and datasets and confirm experimentally that it can significantly improve the cost--accuracy tradeoff compared with uniform precision accumulation baselines.