# BinaryRelax: A Relaxation Approach For Training Deep Neural Networks With Quantized Weights

arXiv: https://arxiv.org/abs/1801.06313 · HTML: https://mlsanity.com/paper/1801.06313/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Jan 2018.

## Built on by (most cited first, 24 of 24)

- [A Comprehensive Survey on Model Quantization for Deep Neural Networks in Image Classification](https://mlsanity.com/paper/2205.07877/) (2022)
- [QuPeD: Quantized Personalization via Distillation with Applications to Federated Learning](https://mlsanity.com/paper/2107.13892/) (2021)
- [MagR: Weight Magnitude Reduction for Enhancing Post-Training Quantization](https://mlsanity.com/paper/2406.00800/) (2024)
- [COMQ: A Backpropagation-Free Algorithm for Post-Training Quantization](https://mlsanity.com/paper/2403.07134/) (2024)
- [Demystifying and Generalizing BinaryConnect](https://mlsanity.com/paper/2110.13220/) (2021)
- [Efficient Neural Networks for Tiny Machine Learning: A Comprehensive Review](https://mlsanity.com/paper/2311.11883/) (2023)
- [Feature Affinity Assisted Knowledge Distillation and Quantization of Deep Neural Networks on Label-Free Data](https://mlsanity.com/paper/2302.10899/) (2023)
- [CLoQ: Enhancing Fine-Tuning of Quantized LLMs via Calibrated LoRA Initialization](https://mlsanity.com/paper/2501.18475/) (2025)
- [PARQ: Piecewise-Affine Regularized Quantization](https://mlsanity.com/paper/2503.15748/) (2025)
- [High-Dimensional Learning Dynamics of Quantized Models with Straight-Through Estimator](https://mlsanity.com/paper/2510.10693/) (2025)
- [QuPeL: Quantized Personalization with Applications to Federated Learning](https://mlsanity.com/paper/2102.11786/) (2021)
- [Model compression as constrained optimization, with application to neural nets. Part V: combining compressions](https://mlsanity.com/paper/2107.04380/) (2021)
- [Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions](https://mlsanity.com/paper/2108.13055/) (2021)
- [RQP-SGD: Differential Private Machine Learning through Noisy SGD and Randomized Quantization](https://mlsanity.com/paper/2402.06606/) (2024)
- [Frame Quantization of Neural Networks](https://mlsanity.com/paper/2404.08131/) (2024)
- [Beyond Discreteness: Sample Complexity Analysis of Straight-Through Estimator for 1-bit Quantization](https://mlsanity.com/paper/2505.18113/) (2025)
- [DNN Quantization with Attention](https://mlsanity.com/paper/2103.13322/) (2021)
- [Channel Pruning In Quantization-aware Training: An Adaptive Projection-gradient Descent-shrinkage-splitting Method](https://mlsanity.com/paper/2204.04375/) (2022)
- [SONIQ: System-Optimized Noise-Injected Ultra-Low-Precision Quantization with Full-Precision Parity](https://mlsanity.com/paper/2311.14114/) (2023)
- [Quality Scalable Quantization Methodology for Deep Learning on Edge](https://mlsanity.com/paper/2407.11260/) (2024)
- [Quantization through Piecewise-Affine Regularization: Optimization and Statistical Guarantees](https://mlsanity.com/paper/2508.11112/) (2025)
- [Soft Quantization: Model Compression Via Weight Coupling](https://mlsanity.com/paper/2601.21219/) (2026)
- [Layerwise Progressive Freezing Enables STE-Free Training of Deep Binary Neural Networks](https://mlsanity.com/paper/2601.22660/) (2026)
- [Layerwise Progressive Freezing: A Training Scaffold for Depth-Scalable Binary Networks](https://mlsanity.com/paper/2606.27759/) (2026)

## Abstract (arXiv)

We propose BinaryRelax, a simple two-phase algorithm, for training deep neural networks with quantized weights. The set constraint that characterizes the quantization of weights is not imposed until the late stage of training, and a sequence of pseudo quantized weights is maintained. Specifically, we relax the hard constraint into a continuous regularizer via Moreau envelope, which turns out to be the squared Euclidean distance to the set of quantized weights. The pseudo quantized weights are obtained by linearly interpolating between the float weights and their quantizations. A continuation strategy is adopted to push the weights towards the quantized state by gradually increasing the regularization parameter. In the second phase, exact quantization scheme with a small learning rate is invoked to guarantee fully quantized weights. We test BinaryRelax on the benchmark CIFAR and ImageNet color image datasets to demonstrate the superiority of the relaxed quantization approach and the improved accuracy over the state-of-the-art training methods. Finally, we prove the convergence of BinaryRelax under an approximate orthogonality condition.