# Recurrence of Optimum for Training Weight and Activation Quantized Networks

arXiv: https://arxiv.org/abs/2012.05529 · HTML: https://mlsanity.com/paper/2012.05529/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Dec 2020.

## Built on by (most cited first, 3 of 3)

- [Feature Affinity Assisted Knowledge Distillation and Quantization of Deep Neural Networks on Label-Free Data](https://mlsanity.com/paper/2302.10899/) (2023)
- [Frame Quantization of Neural Networks](https://mlsanity.com/paper/2404.08131/) (2024)
- [Beyond Discreteness: Sample Complexity Analysis of Straight-Through Estimator for 1-bit Quantization](https://mlsanity.com/paper/2505.18113/) (2025)

## Abstract (arXiv)

Deep neural networks (DNNs) are quantized for efficient inference on resource-constrained platforms. However, training deep learning models with low-precision weights and activations involves a demanding optimization task, which calls for minimizing a stage-wise loss function subject to a discrete set-constraint. While numerous training methods have been proposed, existing studies for full quantization of DNNs are mostly empirical. From a theoretical point of view, we study practical techniques for overcoming the combinatorial nature of network quantization. Specifically, we investigate a simple yet powerful projected gradient-like algorithm for quantizing two-linear-layer networks, which proceeds by repeatedly moving one step at float weights in the negation of a heuristic fake gradient of the loss function (so-called coarse gradient) evaluated at quantized weights. For the first time, we prove that under mild conditions, the sequence of quantized weights recurrently visits the global optimum of the discrete minimization problem for training fully quantized network. We also show numerical evidence of the recurrence phenomenon of weight evolution in training quantized deep networks.