# Is Integer Arithmetic Enough for Deep Learning Training?

arXiv: https://arxiv.org/abs/2207.08822 · HTML: https://mlsanity.com/paper/2207.08822/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 33 references that are in the map.
- First posted on arXiv 18 Jul 2022.

## Builds on (most cited first, 16 of 16)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [MMDetection: Open MMLab Detection Toolbox and Benchmark](https://mlsanity.com/paper/1906.07155/) (2019)
- [Optimization Methods for Large-Scale Machine Learning](https://mlsanity.com/paper/1606.04838/) (2016)
- [Scalable Methods for 8-bit Training of Neural Networks](https://mlsanity.com/paper/1805.11046/) (2018)
- [Towards Unified INT8 Training for Convolutional Neural Network](https://mlsanity.com/paper/1912.12607/) (2019)
- [Training DNNs with Hybrid Block Floating Point](https://mlsanity.com/paper/1804.01526/) (2018)
- [Distribution Adaptive INT8 Quantization for Training CNNs](https://mlsanity.com/paper/2102.04782/) (2021)
- [F8Net: Fixed-Point 8-bit Only Multiplication for Network Quantization](https://mlsanity.com/paper/2202.05239/) (2022)
- [NITI: Training Integer Neural Networks Using Integer-only Arithmetic](https://mlsanity.com/paper/2009.13108/) (2020)
- [Per-Tensor Fixed-Point Quantization of the Back-Propagation Algorithm](https://mlsanity.com/paper/1812.11732/) (2018)

## Built on by (most cited first, 12 of 12)

- [On the Convergence of Stochastic Gradient Descent in Low-precision Number Formats](https://mlsanity.com/paper/2301.01651/) (2023)
- [Understanding Neural Network Binarization with Forward and Backward Proximal Quantizers](https://mlsanity.com/paper/2402.17710/) (2024)
- [Towards Fine-tuning Pre-trained Language Models with Integer Forward and Backward Propagation](https://mlsanity.com/paper/2209.09815/) (2022)
- [Mitigating Outlier Activations in Low-Precision Fine-Tuning of Language Models](https://mlsanity.com/paper/2312.09211/) (2023)
- [Towards Accurate and Efficient Sub-8-Bit Integer Training](https://mlsanity.com/paper/2411.10948/) (2024)
- [HOT: Hadamard-based Optimized Training](https://mlsanity.com/paper/2503.21261/) (2025)
- [AdpQ: A Zero-shot Calibration Free Adaptive Post Training Quantization Method for LLMs](https://mlsanity.com/paper/2405.13358/) (2024)
- [NITRO-D: Native Integer-only Training of Deep Convolutional Neural Networks](https://mlsanity.com/paper/2407.11698/) (2024)
- [Virtualization of Tiny Embedded Systems with a robust real-time capable and extensible Stack Virtual Machine REXAVM supporting Material-integrated Intelligent Systems and Tiny Machine Learning](https://mlsanity.com/paper/2302.09002/) (2023)
- [Mathematical Challenges in Deep Learning](https://mlsanity.com/paper/2303.15464/) (2023)
- [Bitwidth-Specific Logarithmic Arithmetic for Future Hardware-Accelerated Training](https://mlsanity.com/paper/2510.17058/) (2025)
- [Memory-Efficient Integer-Only Training of Deep Neural Networks](https://mlsanity.com/paper/2610.16298/) (2026)

## Abstract (arXiv)

The ever-increasing computational complexity of deep learning models makes their training and deployment difficult on various cloud and edge platforms. Replacing floating-point arithmetic with low-bit integer arithmetic is a promising approach to save energy, memory footprint, and latency of deep learning models. As such, quantization has attracted the attention of researchers in recent years. However, using integer numbers to form a fully functional integer training pipeline including forward pass, back-propagation, and stochastic gradient descent is not studied in detail. Our empirical and mathematical results reveal that integer arithmetic seems to be enough to train deep learning models. Unlike recent proposals, instead of quantization, we directly switch the number representation of computations. Our novel training method forms a fully integer training pipeline that does not change the trajectory of the loss and accuracy compared to floating-point, nor does it need any special hyper-parameter tuning, distribution adjustment, or gradient clipping. Our experimental results show that our proposed method is effective in a wide variety of tasks such as classification (including vision transformers), object detection, and semantic segmentation.