mlsanityData to 14 Oct 2026

A Statistical Framework for Low-bitwidth Training of Deep Neural Networks

arXiv 2010.14298 ↗First posted 27 Oct 2020cs.LG stat.MLcited by 51

Not on the map. Why?

Abstract (arXiv)

Fully quantized training (FQT), which uses low-bitwidth hardware by quantizing the activations, weights, and gradients of a neural network model, is a promising approach to accelerate the training of deep neural networks. One major challenge with FQT is the lack of theoretical understanding, in particular of how gradient quantization impacts convergence properties. In this paper, we address this problem by presenting a statistical framework for analyzing FQT algorithms. We view the quantized gradient of FQT as a stochastic estimator of its full precision counterpart, a procedure known as quantization-aware training (QAT). We show that the FQT gradient is an unbiased estimator of the QAT gradient, and we discuss the impact of gradient quantization on its variance. Inspired by these theoretical results, we develop two novel gradient quantizers, and we show that these have smaller variance than the existing per-tensor quantizer. For training ResNet-50 on ImageNet, our 5-bit block Householder quantizer achieves only 0.5% validation accuracy loss relative to QAT, comparable to the existing INT8 baseline.

Key facts. Cited by 51 papers in the map (data through 14 Oct 2026). First posted on arXiv 27 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 6; 2022: 8; 2023: 10; 2024: 11; 2025: 11; 2026: 5.

51 papers.

Sort:

By area:

  1. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  2. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 815
  3. 2025SageAttention3: Microscaling FP4 Attention for Inference and An Exploration of 8-Bit Trainingcited by 66
  4. 2021ActNN: Reducing Training Memory Footprint via 2-Bit Activation Compressed Trainingcited by 61
  5. 2023Training Transformers with 4-bit Integerscited by 60
  6. 2024COAT: Compressing Optimizer states and Activation for Memory-Efficient FP8 Trainingcited by 33
  7. 2022GACT: Activation Compressed Training for Generic Network Architecturescited by 31
  8. 2024Jetfire: Efficient and Accurate Transformer Pretraining with INT8 Data Flow and Per-Block Quantizationcited by 31
  9. 2022F8Net: Fixed-Point 8-bit Only Multiplication for Network Quantizationcited by 28
  10. 2021Accurate Neural Training with 4-bit Matrix Multiplications at Standard Formatscited by 26
  11. 2023Watermarking LLMs with Weight Quantizationcited by 22
  12. 2025Oscillation-Reduced MXFP4 Training for Vision Transformerscited by 21
38 more
  1. 2025Accurate INT8 Training Through Dynamic Block-Level Fallbackcited by 14
  2. 2025TetraJet-v2: Accurate NVFP4 Training for Large Language Models with Oscillation Suppression and Outlier Controlcited by 14
  3. 2023Efficient On-device Training via Gradient Filteringcited by 13
  4. 2025Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunitiescited by 11
  5. 2023Adaptive Message Quantization and Parallelization for Distributed Full-graph GNN Trainingcited by 10
  6. 2023Efficient Low-rank Backpropagation for Vision Transformer Adaptationcited by 10
  7. 2024Collage: Light-Weight Low-Precision Strategy for LLM Trainingcited by 10
  8. 2023Design Principles for Lifelong Learning AI Acceleratorscited by 9
  9. 2023SPIDE: A Purely Spike-based Method for Training Feedback Spiking Neural Networkscited by 8
  10. 2025End-to-end workflow for machine learning-based qubit readout with QICK and hls4mlcited by 8
  11. 2026SageBwd: A Trainable Low-bit Attentioncited by 8
  12. 2021In-Hindsight Quantization Range Estimation for Quantized Trainingcited by 7
  13. 2021LNS-Madam: Low-Precision Training in Logarithmic Number System using Multiplicative Weight Updatecited by 7
  14. 2022Quantized Training of Gradient Boosting Decision Treescited by 6
  15. 2022Randomized Quantization: A Generic Augmentation for Data Agnostic Self-supervised Learningcited by 6
  16. 2024Poor Man's Training on MCUs: A Memory-Efficient Quantized Back-Propagation-Free Approachcited by 5
  17. 2022Fast Lossless Neural Compression with Integer-Only Discrete Flowscited by 4
  18. 2022DIVISION: Memory Efficient Training via Dual Activation Precisioncited by 4
  19. 2021Low-Precision Reinforcement Learning: Running Soft Actor-Critic in Half Precisioncited by 3
  20. 2023Hadamard Domain Training with Integers for Class Incremental Quantized Learningcited by 3
  21. 2024BOLD: Boolean Logic Deep Learningcited by 3
  22. 20241-Bit FQT: Pushing the Limit of Fully Quantized Training to 1-bitcited by 3
  23. 2022AskewSGD : An Annealed interval-constrained Optimisation method to train Quantized Neural Networkscited by 2
  24. 2023End-to-end codesign of Hessian-aware quantized neural networks for FPGAs and ASICscited by 2
  25. 2023Boolean Variation and Boolean Logic BackPropagationcited by 2
  26. 2024Center-Sensitive Kernel Optimization for Efficient On-Device Incremental Learningcited by 2
  27. 2024HLQ: Fast and Efficient Backpropagation via Hadamard Low-rank Quantizationcited by 2
  28. 2024Towards Accurate and Efficient Sub-8-Bit Integer Trainingcited by 2
  29. 2025HOT: Hadamard-based Optimized Trainingcited by 2
  30. 2024Toward INT4 Fixed-Point Training via Exploring Quantization Error for Gradientscited by 1
  31. 2025Coded Deep Learning: Framework and Algorithmcited by 1
  32. 2025QuantU-Net: Efficient Wearable Medical Imaging Using Bitwidth as a Trainable Parametercited by 1
  33. 2025Training with Fewer Bits: Unlocking Edge LLMs Training with Stochastic Roundingcited by 1
  34. 2025Advances and Challenges in Semantic Textual Similarity: A Comprehensive Surveycited by 1
  35. 2026Every Bit Counts: A Theoretical Study of Precision-Expressivity Tradeoffs in Quantized Transformerscited by 1
  36. 2026BinaryAttention: One-Bit QK-Attention for Vision and Diffusion Transformerscited by 1
  37. 2026Real-Time Detection of Charge Jumps in Superconducting Qubits with a Convolutional Neural Networkcited by 1
  38. 2026Why Does Post-Training Quantization Work?cited by 1

50 most cited of 51 All 51 (JSON)