# Training Quantized Nets: A Deeper Understanding

arXiv: https://arxiv.org/abs/1706.02379 · HTML: https://mlsanity.com/paper/1706.02379/

## Key facts

- Cited by 75 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Jun 2017.

## Built on by (most cited first, 30 of 75)

- [Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discovery](https://mlsanity.com/paper/2302.03668/) (2023)
- [PokeBNN: A Binary Pursuit of Lightweight Accuracy](https://mlsanity.com/paper/2112.00133/) (2021)
- [DAdaQuant: Doubly-adaptive quantization for communication-efficient Federated Learning](https://mlsanity.com/paper/2111.00465/) (2021)
- [Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradients](https://mlsanity.com/paper/2407.08296/) (2024)
- [QuPeD: Quantized Personalization via Distillation with Applications to Federated Learning](https://mlsanity.com/paper/2107.13892/) (2021)
- [AdaRankGrad: Adaptive Gradient-Rank and Moments for Memory-Efficient LLMs Training and Fine-Tuning](https://mlsanity.com/paper/2410.17881/) (2024)
- [Wavelet Feature Maps Compression for Image-to-Image CNNs](https://mlsanity.com/paper/2205.12268/) (2022)
- [Secure Quantized Training for Deep Learning](https://mlsanity.com/paper/2107.00501/) (2021)
- [Low-Precision Stochastic Gradient Langevin Dynamics](https://mlsanity.com/paper/2206.09909/) (2022)
- [Quantization Robust Federated Learning for Efficient Inference on Heterogeneous Devices](https://mlsanity.com/paper/2206.10844/) (2022)
- [Experimental Analysis of Large-scale Learnable Vector Storage Compression](https://mlsanity.com/paper/2311.15578/) (2023)
- [Adapting by Pruning: A Case Study on BERT](https://mlsanity.com/paper/2105.03343/) (2021)
- [Demystifying and Generalizing BinaryConnect](https://mlsanity.com/paper/2110.13220/) (2021)
- [Adaptive Low-Precision Training for Embeddings in Click-Through Rate Prediction](https://mlsanity.com/paper/2212.05735/) (2022)
- [Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunities](https://mlsanity.com/paper/2505.01043/) (2025)
- [Stochastic Approximation Beyond Gradient for Signal Processing and Machine Learning](https://mlsanity.com/paper/2302.11147/) (2023)
- [Collage: Light-Weight Low-Precision Strategy for LLM Training](https://mlsanity.com/paper/2405.03637/) (2024)
- [Stochastic Rounding for LLM Training: Theory and Practice](https://mlsanity.com/paper/2502.20566/) (2025)
- [Service Delay Minimization for Federated Learning over Mobile Devices](https://mlsanity.com/paper/2205.09868/) (2022)
- [Adaptive Precision Training (AdaPT): A dynamic fixed point quantized training approach for DNNs](https://mlsanity.com/paper/2107.13490/) (2021)
- [CoCoFL: Communication- and Computation-Aware Federated Learning via Partial NN Freezing and Quantization](https://mlsanity.com/paper/2203.05468/) (2022)
- [Reducing Computational Complexity of Neural Networks in Optical Channel Equalization: From Concepts to Implementation](https://mlsanity.com/paper/2208.12866/) (2022)
- [QGen: On the Ability to Generalize in Quantization Aware Training](https://mlsanity.com/paper/2404.11769/) (2024)
- [OAC: Output-adaptive Calibration for Accurate Post-training Quantization](https://mlsanity.com/paper/2405.15025/) (2024)
- [Random and Adversarial Bit Error Robustness: Energy-Efficient and Secure DNN Accelerators](https://mlsanity.com/paper/2104.08323/) (2021)
- [Update Compression for Deep Neural Networks on the Edge](https://mlsanity.com/paper/2203.04516/) (2022)
- [Quantized Training of Gradient Boosting Decision Trees](https://mlsanity.com/paper/2207.09682/) (2022)
- [Multiplication-Free Transformer Training via Piecewise Affine Operations](https://mlsanity.com/paper/2305.17190/) (2023)
- [RoSTE: An Efficient Quantization-Aware Supervised Fine-Tuning Approach for Large Language Models](https://mlsanity.com/paper/2502.09003/) (2025)
- [LORENZA: Enhancing Generalization in Low-Rank Gradient LLM Training via Efficient Zeroth-Order Adaptive SAM](https://mlsanity.com/paper/2502.19571/) (2025)

## Abstract (arXiv)

Currently, deep neural networks are deployed on low-power portable devices by first training a full-precision model using powerful hardware, and then deriving a corresponding low-precision model for efficient inference on such systems. However, training models directly with coarsely quantized weights is a key step towards learning on embedded platforms that have limited computing resources, memory capacity, and power consumption. Numerous recent publications have studied methods for training quantized networks, but these studies have mostly been empirical. In this work, we investigate training methods for quantized neural networks from a theoretical viewpoint. We first explore accuracy guarantees for training methods under convexity assumptions. We then look at the behavior of these algorithms for non-convex problems, and show that training algorithms that exploit high-precision representations have an important greedy search phase that purely quantized training methods lack, which explains the difficulty of training using low-precision arithmetic.