# Quantized Neural Networks for Low-Precision Accumulation with Guaranteed Overflow Avoidance

arXiv: https://arxiv.org/abs/2301.13376 · HTML: https://mlsanity.com/paper/2301.13376/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 40 references that are in the map.
- First posted on arXiv 31 Jan 2023.

## Builds on (most cited first, 28 of 28)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network](https://mlsanity.com/paper/1609.05158/) (2016)
- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks](https://mlsanity.com/paper/1602.07868/) (2016)
- [Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks](https://mlsanity.com/paper/2102.00554/) (2021)
- [A White Paper on Neural Network Quantization](https://mlsanity.com/paper/2106.08295/) (2021)
- [Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations](https://mlsanity.com/paper/1609.07061/) (2016)
- [Data-Free Quantization Through Weight Equalization and Bias Correction](https://mlsanity.com/paper/1906.04721/) (2019)
- [Training Deep Neural Networks with 8-bit Floating Point Numbers](https://mlsanity.com/paper/1812.08011/) (2018)
- [Soft-to-Hard Vector Quantization for End-to-End Learning Compressible Representations](https://mlsanity.com/paper/1704.00648/) (2017)
- [FINN: A Framework for Fast, Scalable Binarized Neural Network Inference](https://mlsanity.com/paper/1612.07119/) (2016)
- [Sparse GPU Kernels for Deep Learning](https://mlsanity.com/paper/2006.10901/) (2020)
- [Trained Quantization Thresholds for Accurate and Efficient Fixed-Point Inference of Deep Neural Networks](https://mlsanity.com/paper/1903.08066/) (2019)
- [FINN-R: An End-to-End Deep-Learning Framework for Fast Exploration of Quantized Neural Networks](https://mlsanity.com/paper/1809.04570/) (2018)
- [SHE: A Fast and Accurate Deep Neural Network for Encrypted Data](https://mlsanity.com/paper/1906.00148/) (2019)
- [Direct Behavior Specification via Constrained Reinforcement Learning](https://mlsanity.com/paper/2112.12228/) (2021)
- [Directional Pruning of Deep Neural Networks](https://mlsanity.com/paper/2006.09358/) (2020)
- [Accumulation Bit-Width Scaling For Ultra-Low Precision Training Of Deep Networks](https://mlsanity.com/paper/1901.06588/) (2019)
- [Quantization of Deep Neural Networks for Accumulator-constrained Processors](https://mlsanity.com/paper/2004.11783/) (2020)
- [Weight Normalization based Quantization for Deep Neural Network Compression](https://mlsanity.com/paper/1907.00593/) (2019)
- [Compressibility Loss for Neural Network Weights](https://mlsanity.com/paper/1905.01044/) (2019)
- [An Energy-Efficient Edge Computing Paradigm for Convolution-based Image Upsampling](https://mlsanity.com/paper/2107.07647/) (2021)

## Built on by (most cited first, 3 of 3)

- [Accumulator-Aware Post-Training Quantization for Large Language Models](https://mlsanity.com/paper/2409.17092/) (2024)
- [Mixed precision accumulation for neural network inference guided by componentwise forward error analysis](https://mlsanity.com/paper/2503.15568/) (2025)
- [Quantizing Recursive Reasoning Models](https://mlsanity.com/paper/2607.16237/) (2026)

## Abstract (arXiv)

We introduce a quantization-aware training algorithm that guarantees avoiding numerical overflow when reducing the precision of accumulators during inference. We leverage weight normalization as a means of constraining parameters during training using accumulator bit width bounds that we derive. We evaluate our algorithm across multiple quantized models that we train for different tasks, showing that our approach can reduce the precision of accumulators while maintaining model accuracy with respect to a floating-point baseline. We then show that this reduction translates to increased design efficiency for custom FPGA-based accelerators. Finally, we show that our algorithm not only constrains weights to fit into an accumulator of user-defined bit width, but also increases the sparsity and compressibility of the resulting weights. Across all of our benchmark models trained with 8-bit weights and activations, we observe that constraining the hidden layers of quantized neural networks to fit into 16-bit accumulators yields an average 98.2% sparsity with an estimated compression rate of 46.5x all while maintaining 99.2% of the floating-point performance.