# Mixed Precision Training of Convolutional Neural Networks using Integer Operations

arXiv: https://arxiv.org/abs/1802.00930 · HTML: https://mlsanity.com/paper/1802.00930/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Feb 2018.

## Built on by (most cited first, 30 of 45)

- [Pruning and Quantization for Deep Neural Network Acceleration: A Survey](https://mlsanity.com/paper/2101.09671/) (2021)
- [Zero-shot Adversarial Quantization](https://mlsanity.com/paper/2103.15263/) (2021)
- [A Survey on Efficient Training of Transformers](https://mlsanity.com/paper/2302.01107/) (2023)
- [An Efficient Statistical-based Gradient Compression Technique for Distributed Training Systems](https://mlsanity.com/paper/2101.10761/) (2021)
- [8-bit Numerical Formats for Deep Neural Networks](https://mlsanity.com/paper/2206.02915/) (2022)
- [Exploring System Performance of Continual Learning for Mobile and Embedded Sensing Applications](https://mlsanity.com/paper/2110.13290/) (2021)
- [Data Quality-aware Mixed-precision Quantization via Hybrid Reinforcement Learning](https://mlsanity.com/paper/2302.04453/) (2023)
- [Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunities](https://mlsanity.com/paper/2505.01043/) (2025)
- [GradPIM: A Practical Processing-in-DRAM Architecture for Gradient Descent](https://mlsanity.com/paper/2102.07511/) (2021)
- [Low-Precision Arithmetic for Fast Gaussian Processes](https://mlsanity.com/paper/2207.06856/) (2022)
- [BigVideo: A Large-scale Video Subtitle Translation Dataset for Multimodal Machine Translation](https://mlsanity.com/paper/2305.18326/) (2023)
- [Integer-arithmetic-only Certified Robustness for Quantized Neural Networks](https://mlsanity.com/paper/2108.09413/) (2021)
- [Designing Large Foundation Models for Efficient Training and Inference: A Survey](https://mlsanity.com/paper/2409.01990/) (2024)
- [Lumos: Efficient Performance Modeling and Estimation for Large-scale LLM Training](https://mlsanity.com/paper/2504.09307/) (2025)
- [HR-INR: Continuous Space-Time Video Super-Resolution via Event Camera](https://mlsanity.com/paper/2405.13389/) (2024)
- [All-You-Can-Fit 8-Bit Flexible Floating-Point Format for Accurate and Memory-Efficient Inference of Deep Neural Networks](https://mlsanity.com/paper/2104.07329/) (2021)
- [Rethinking Floating Point Overheads for Mixed Precision DNN Accelerators](https://mlsanity.com/paper/2101.11748/) (2021)
- [Towards Efficient Full 8-bit Integer DNN Online Training on Resource-limited Devices without Batch Normalization](https://mlsanity.com/paper/2105.13890/) (2021)
- [Combining Compressions for Multiplicative Size Scaling on Natural Language Tasks](https://mlsanity.com/paper/2208.09684/) (2022)
- [Number Systems for Deep Neural Network Architectures: A Survey](https://mlsanity.com/paper/2307.05035/) (2023)
- [Tango: rethinking quantization for graph neural network training on GPUs](https://mlsanity.com/paper/2308.00890/) (2023)
- [Enabling On-device Continual Learning with Binary Neural Networks](https://mlsanity.com/paper/2401.09916/) (2024)
- [Low-Precision Reinforcement Learning: Running Soft Actor-Critic in Half Precision](https://mlsanity.com/paper/2102.13565/) (2021)
- [MERCURY: Accelerating DNN Training By Exploiting Input Similarity](https://mlsanity.com/paper/2110.14904/) (2021)
- [BitTrain: Sparse Bitmap Compression for Memory-Efficient Training on the Edge](https://mlsanity.com/paper/2110.15362/) (2021)
- [Accuracy Booster: Enabling 4-bit Fixed-point Arithmetic for DNN Training](https://mlsanity.com/paper/2211.10737/) (2022)
- [On-Device Learning with Binary Neural Networks](https://mlsanity.com/paper/2308.15308/) (2023)
- [1-Bit FQT: Pushing the Limit of Fully Quantized Training to 1-bit](https://mlsanity.com/paper/2408.14267/) (2024)
- [Low-Precision Streaming PCA](https://mlsanity.com/paper/2510.22440/) (2025)
- [Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions](https://mlsanity.com/paper/2108.13055/) (2021)

## Abstract (arXiv)

The state-of-the-art (SOTA) for mixed precision training is dominated by variants of low precision floating point operations, and in particular, FP16 accumulating into FP32 Micikevicius et al. (2017). On the other hand, while a lot of research has also happened in the domain of low and mixed-precision Integer training, these works either present results for non-SOTA networks (for instance only AlexNet for ImageNet-1K), or relatively small datasets (like CIFAR-10). In this work, we train state-of-the-art visual understanding neural networks on the ImageNet-1K dataset, with Integer operations on General Purpose (GP) hardware. In particular, we focus on Integer Fused-Multiply-and-Accumulate (FMA) operations which take two pairs of INT16 operands and accumulate results into an INT32 output.We propose a shared exponent representation of tensors and develop a Dynamic Fixed Point (DFP) scheme suitable for common neural network operations. The nuances of developing an efficient integer convolution kernel is examined, including methods to handle overflow of the INT32 accumulator. We implement CNN training for ResNet-50, GoogLeNet-v1, VGG-16 and AlexNet; and these networks achieve or exceed SOTA accuracy within the same number of iterations as their FP32 counterparts without any change in hyper-parameters and with a 1.8X improvement in end-to-end training throughput. To the best of our knowledge these results represent the first INT16 training results on GP hardware for ImageNet-1K dataset using SOTA CNNs and achieve highest reported accuracy using half-precision