# Exploring the Potential of Low-bit Training of Convolutional Neural Networks

arXiv: https://arxiv.org/abs/2006.02804 · HTML: https://mlsanity.com/paper/2006.02804/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Jun 2020.

## Built on by (most cited first, 7 of 7)

- [Accurate Neural Training with 4-bit Matrix Multiplications at Standard Formats](https://mlsanity.com/paper/2112.10769/) (2021)
- [Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunities](https://mlsanity.com/paper/2505.01043/) (2025)
- [Towards Accurate and Efficient Sub-8-Bit Integer Training](https://mlsanity.com/paper/2411.10948/) (2024)
- [Toward INT4 Fixed-Point Training via Exploring Quantization Error for Gradients](https://mlsanity.com/paper/2407.12637/) (2024)
- [Distribution-Aware Hadamard Quantization for Hardware-Efficient Implicit Neural Representations](https://mlsanity.com/paper/2508.13478/) (2025)
- [Block-Wise Dynamic-Precision Neural Network Training Acceleration via Online Quantization Sensitivity Analytics](https://mlsanity.com/paper/2210.17047/) (2022)
- [FedProphet: Memory-Efficient Federated Adversarial Training via Robust and Consistent Cascade Learning](https://mlsanity.com/paper/2409.08372/) (2024)

## Abstract (arXiv)

In this work, we propose a low-bit training framework for convolutional neural networks, which is built around a novel multi-level scaling (MLS) tensor format. Our framework focuses on reducing the energy consumption of convolution operations by quantizing all the convolution operands to low bit-width format. Specifically, we propose the MLS tensor format, in which the element-wise bit-width can be largely reduced. Then, we describe the dynamic quantization and the low-bit tensor convolution arithmetic to leverage the MLS tensor format efficiently. Experiments show that our framework achieves a superior trade-off between the accuracy and the bit-width than previous low-bit training frameworks. For training a variety of models on CIFAR-10, using 1-bit mantissa and 2-bit exponent is adequate to keep the accuracy loss within 1%. And on larger datasets like ImageNet, using 4-bit mantissa and 2-bit exponent is adequate to keep the accuracy loss within 1%. Through the energy consumption simulation of the computing units, we can estimate that training a variety of models with our framework could achieve 8.3~10.2× and 1.9~2.3× higher energy efficiency than training with full-precision and 8-bit floating-point arithmetic, respectively.