# Learning low-precision neural networks without Straight-Through Estimator(STE)

arXiv: https://arxiv.org/abs/1903.01061 · HTML: https://mlsanity.com/paper/1903.01061/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Mar 2019.

## Built on by (most cited first, 16 of 16)

- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [Pruning and Quantization for Deep Neural Network Acceleration: A Survey](https://mlsanity.com/paper/2101.09671/) (2021)
- [Differentiable Model Compression via Pseudo Quantization Noise](https://mlsanity.com/paper/2104.09987/) (2021)
- [Learnable Companding Quantization for Accurate Low-bit Neural Networks](https://mlsanity.com/paper/2103.07156/) (2021)
- [Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safety](https://mlsanity.com/paper/2104.14235/) (2021)
- [Compiler-Aware Neural Architecture Search for On-Mobile Real-time Super-Resolution](https://mlsanity.com/paper/2207.12577/) (2022)
- [Patch-wise Mixed-Precision Quantization of Vision Transformer](https://mlsanity.com/paper/2305.06559/) (2023)
- [QCore: Data-Efficient, On-Device Continual Calibration for Quantized Models -- Extended Version](https://mlsanity.com/paper/2404.13990/) (2024)
- [A Low Memory Footprint Quantized Neural Network for Depth Completion of Very Sparse Time-of-Flight Depth Maps](https://mlsanity.com/paper/2205.12918/) (2022)
- [All-in-One: A Highly Representative DNN Pruning Framework for Edge Devices with Dynamic Power Management](https://mlsanity.com/paper/2212.05122/) (2022)
- [QFT: Post-training quantization via fast joint finetuning of all degrees of freedom](https://mlsanity.com/paper/2212.02634/) (2022)
- [One-Step Forward and Backtrack: Overcoming Zig-Zagging in Loss-Aware Quantization Training](https://mlsanity.com/paper/2401.16760/) (2024)
- [Quantization of Neural Network Equalizers in Optical Fiber Transmission Experiments](https://mlsanity.com/paper/2310.05950/) (2023)
- [Reclaiming Residual Knowledge: A Novel Paradigm to Low-Bit Quantization](https://mlsanity.com/paper/2408.00923/) (2024)
- [Layerwise Progressive Freezing Enables STE-Free Training of Deep Binary Neural Networks](https://mlsanity.com/paper/2601.22660/) (2026)
- [Layerwise Progressive Freezing: A Training Scaffold for Depth-Scalable Binary Networks](https://mlsanity.com/paper/2606.27759/) (2026)

## Abstract (arXiv)

The Straight-Through Estimator (STE) is widely used for back-propagating gradients through the quantization function, but the STE technique lacks a complete theoretical understanding. We propose an alternative methodology called alpha-blending (AB), which quantizes neural networks to low-precision using stochastic gradient descent (SGD). Our method (AB) avoids STE approximation by replacing the quantized weight in the loss function by an affine combination of the quantized weight wq and the corresponding full-precision weight w with non-trainable scalar coefficient α and 1-α. During training, α is gradually increased from 0 to 1; the gradient updates to the weights are through the full-precision term, (1-α)w, of the affine combination; the model is converted from full-precision to low-precision progressively. To evaluate the method, a 1-bit BinaryNet on CIFAR10 dataset and 8-bits, 4-bits MobileNet v1, ResNet50 v1/2 on ImageNet dataset are trained using the alpha-blending approach, and the evaluation indicates that AB improves top-1 accuracy by 0.9%, 0.82% and 2.93% respectively compared to the results of STE based quantization.