# Training High-Performance and Large-Scale Deep Neural Networks with Full 8-bit Integers

arXiv: https://arxiv.org/abs/1909.02384 · HTML: https://mlsanity.com/paper/1909.02384/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Sep 2019.

## Built on by (most cited first, 30 of 33)

- [Pruning and Quantization for Deep Neural Network Acceleration: A Survey](https://mlsanity.com/paper/2101.09671/) (2021)
- [Training Transformers with 4-bit Integers](https://mlsanity.com/paper/2306.11987/) (2023)
- [Optimizing Large Language Model Training Using FP4 Quantization](https://mlsanity.com/paper/2501.17116/) (2025)
- [Quartet: Native FP4 Training Can Be Optimal for Large Language Models](https://mlsanity.com/paper/2505.14669/) (2025)
- [Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradients](https://mlsanity.com/paper/2407.08296/) (2024)
- [FP8 versus INT8 for efficient deep learning inference](https://mlsanity.com/paper/2303.17951/) (2023)
- [Distribution Adaptive INT8 Quantization for Training CNNs](https://mlsanity.com/paper/2102.04782/) (2021)
- [GSQ-Tuning: Group-Shared Exponents Integer in Fully Quantized Training for LLMs On-Device Fine-tuning](https://mlsanity.com/paper/2502.12913/) (2025)
- [CPT: Efficient Deep Neural Network Training via Cyclic Precision](https://mlsanity.com/paper/2101.09868/) (2021)
- [Data Quality-aware Mixed-precision Quantization via Hybrid Reinforcement Learning](https://mlsanity.com/paper/2302.04453/) (2023)
- [SmartFRZ: An Efficient Training Framework using Attention-Based Layer Freezing](https://mlsanity.com/paper/2401.16720/) (2024)
- [Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunities](https://mlsanity.com/paper/2505.01043/) (2025)
- [Resource-Efficient Deep Learning: A Survey on Model-, Arithmetic-, and Implementation-Level Techniques](https://mlsanity.com/paper/2112.15131/) (2021)
- [In-Hindsight Quantization Range Estimation for Quantized Training](https://mlsanity.com/paper/2105.04246/) (2021)
- [Towards Efficient Full 8-bit Integer DNN Online Training on Resource-limited Devices without Batch Normalization](https://mlsanity.com/paper/2105.13890/) (2021)
- [Shift-BNN: Highly-Efficient Probabilistic Bayesian Neural Network Training via Memory-Friendly Pattern Retrieving](https://mlsanity.com/paper/2110.03553/) (2021)
- [1-Bit FQT: Pushing the Limit of Fully Quantized Training to 1-bit](https://mlsanity.com/paper/2408.14267/) (2024)
- [Training for temporal sparsity in deep neural networks, application in video processing](https://mlsanity.com/paper/2107.07305/) (2021)
- [Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions](https://mlsanity.com/paper/2108.13055/) (2021)
- [LDP: Learnable Dynamic Precision for Efficient Deep Neural Network Training and Inference](https://mlsanity.com/paper/2203.07713/) (2022)
- [RCT: Resource Constrained Training for Edge AI](https://mlsanity.com/paper/2103.14493/) (2021)
- [Training Quantized Deep Neural Networks via Cooperative Coevolution](https://mlsanity.com/paper/2112.14834/) (2021)
- [MetaGrad: Adaptive Gradient Quantization with Hypernetworks](https://mlsanity.com/paper/2303.02347/) (2023)
- [Better Schedules for Low Precision Training of Deep Neural Networks](https://mlsanity.com/paper/2403.02243/) (2024)
- [Lightweight Change Detection in Heterogeneous Remote Sensing Images with Online All-Integer Pruning Training](https://mlsanity.com/paper/2405.01920/) (2024)
- [NITRO-D: Native Integer-only Training of Deep Convolutional Neural Networks](https://mlsanity.com/paper/2407.11698/) (2024)
- [Toward INT4 Fixed-Point Training via Exploring Quantization Error for Gradients](https://mlsanity.com/paper/2407.12637/) (2024)
- [Training with Fewer Bits: Unlocking Edge LLMs Training with Stochastic Rounding](https://mlsanity.com/paper/2511.00874/) (2025)
- [SmartDeal: Re-Modeling Deep Network Weights for Efficient Inference and Training](https://mlsanity.com/paper/2101.01163/) (2021)
- [A 14uJ/Decision Keyword Spotting Accelerator with In-SRAM-Computing and On Chip Learning for Customization](https://mlsanity.com/paper/2205.04665/) (2022)

## Abstract (arXiv)

Deep neural network (DNN) quantization converting floating-point (FP) data in the network to integers (INT) is an effective way to shrink the model size for memory saving and simplify the operations for compute acceleration. Recently, researches on DNN quantization develop from inference to training, laying a foundation for the online training on accelerators. However, existing schemes leaving batch normalization (BN) untouched during training are mostly incomplete quantization that still adopts high precision FP in some parts of the data paths. Currently, there is no solution that can use only low bit-width INT data during the whole training process of large-scale DNNs with acceptable accuracy. In this work, through decomposing all the computation steps in DNNs and fusing three special quantization functions to satisfy the different precision requirements, we propose a unified complete quantization framework termed as ``WAGEUBN'' to quantize DNNs involving all data paths including W (Weights), A (Activation), G (Gradient), E (Error), U (Update), and BN. Moreover, the Momentum optimizer is also quantized to realize a completely quantized framework. Experiments on ResNet18/34/50 models demonstrate that WAGEUBN can achieve competitive accuracy on the ImageNet dataset. For the first time, the study of quantization in large-scale DNNs is advanced to the full 8-bit INT level. In this way, all the operations in the training and inference can be bit-wise operations, pushing towards faster processing speed, decreased memory cost, and higher energy efficiency. Our throughout quantization framework has great potential for future efficient portable devices with online learning ability.