# Per-Tensor Fixed-Point Quantization of the Back-Propagation Algorithm

arXiv: https://arxiv.org/abs/1812.11732 · HTML: https://mlsanity.com/paper/1812.11732/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Dec 2018.

## Built on by (most cited first, 15 of 15)

- [BSQ: Exploring Bit-Level Sparsity for Mixed-Precision Neural Network Quantization](https://mlsanity.com/paper/2102.10462/) (2021)
- [Accurate Neural Training with 4-bit Matrix Multiplications at Standard Formats](https://mlsanity.com/paper/2112.10769/) (2021)
- [Optimal Clipping and Magnitude-aware Differentiation for Improved Quantization-aware Training](https://mlsanity.com/paper/2206.06501/) (2022)
- [ESPACE: Dimensionality Reduction of Activations for Model Compression](https://mlsanity.com/paper/2410.05437/) (2024)
- [Is Integer Arithmetic Enough for Deep Learning Training?](https://mlsanity.com/paper/2207.08822/) (2022)
- [Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunities](https://mlsanity.com/paper/2505.01043/) (2025)
- [Collage: Light-Weight Low-Precision Strategy for LLM Training](https://mlsanity.com/paper/2405.03637/) (2024)
- [In-Hindsight Quantization Range Estimation for Quantized Training](https://mlsanity.com/paper/2105.04246/) (2021)
- [LNS-Madam: Low-Precision Training in Logarithmic Number System using Multiplicative Weight Update](https://mlsanity.com/paper/2106.13914/) (2021)
- [Tango: rethinking quantization for graph neural network training on GPUs](https://mlsanity.com/paper/2308.00890/) (2023)
- [MinUn: Accurate ML Inference on Microcontrollers](https://mlsanity.com/paper/2210.16556/) (2022)
- [Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions](https://mlsanity.com/paper/2108.13055/) (2021)
- [AskewSGD : An Annealed interval-constrained Optimisation method to train Quantized Neural Networks](https://mlsanity.com/paper/2211.03741/) (2022)
- [Training with Mixed-Precision Floating-Point Assignments](https://mlsanity.com/paper/2301.13464/) (2023)
- [Error-aware Quantization through Noise Tempering](https://mlsanity.com/paper/2212.05603/) (2022)

## Abstract (arXiv)

The high computational and parameter complexity of neural networks makes their training very slow and difficult to deploy on energy and storage-constrained computing systems. Many network complexity reduction techniques have been proposed including fixed-point implementation. However, a systematic approach for designing full fixed-point training and inference of deep neural networks remains elusive. We describe a precision assignment methodology for neural network training in which all network parameters, i.e., activations and weights in the feedforward path, gradients and weight accumulators in the feedback path, are assigned close to minimal precision. The precision assignment is derived analytically and enables tracking the convergence behavior of the full precision training, known to converge a priori. Thus, our work leads to a systematic methodology of determining suitable precision for fixed-point training. The near optimality (minimality) of the resulting precision assignment is validated empirically for four networks on the CIFAR-10, CIFAR-100, and SVHN datasets. The complexity reduction arising from our approach is compared with other fixed-point neural network designs.