AskewSGD : An Annealed interval-constrained Optimisation method to train Quantized Neural Networks
Not on the map. Why?
Abstract (arXiv)
In this paper, we develop a new algorithm, Annealed Skewed SGD - AskewSGD - for training deep neural networks (DNNs) with quantized weights. First, we formulate the training of quantized neural networks (QNNs) as a smoothed sequence of interval-constrained optimization problems. Then, we propose a new first-order stochastic method, AskewSGD, to solve each constrained optimization subproblem. Unlike algorithms with active sets and feasible directions, AskewSGD avoids projections or optimization under the entire feasible set and allows iterates that are infeasible. The numerical complexity of AskewSGD is comparable to existing approaches for training QNNs, such as the straight-through gradient estimator used in BinaryConnect, or other state of the art methods (ProxQuant, LUQ). We establish convergence guarantees for AskewSGD (under general assumptions for the objective function). Experimental results show that the AskewSGD algorithm performs better than or on par with state of the art methods in classical benchmarks.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 66 of its 94 references that are in the map. First posted on arXiv 7 Nov 2022.
Builds on
66 of 94 references.
References · 66
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
- 2016Wide Residual Networkscited by 3,057
- 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
- 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
- 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,135
- 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 829
- 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
- 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 571
- 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 557
- 2021A White Paper on Neural Network Quantizationcited by 453
- 2018Model compression via distillation and quantizationcited by 287
38 more
- 2018LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networkscited by 272
- 2021Pruning and Quantization for Deep Neural Network Acceleration: A Surveycited by 259
- 2018Bi-Real Net: Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and Advanced Training Algorithmcited by 233
- 2016Trained Ternary Quantizationcited by 203
- 2016Ternary Weight Networkscited by 195
- 2019Low-bit Quantization of Neural Networks for Efficient Inferencecited by 185
- 2020Binary Neural Networks: A Surveycited by 184
- 2020ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functionscited by 183
- 2020LSQ+: Improving low-bit quantization through learnable offsets and better initializationcited by 180
- 2018Training Deep Neural Networks with 8-bit Floating Point Numberscited by 163
- 2019Additive Powers-of-Two Quantization: An Efficient Non-uniform Discretization for Neural Networkscited by 143
- 2018Scalable Methods for 8-bit Training of Neural Networkscited by 139
- 2016Designing Energy-Efficient Convolutional Neural Networks using Energy-Aware Pruningcited by 133
- 2018Stochastic subgradient method converges on tame functionscited by 120
- 2014Neural Variational Inference and Learning in Belief Networkscited by 116
- 2020Training Binary Neural Networks with Real-to-Binary Convolutionscited by 111
- 2017Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracycited by 100
- 2021The Bayesian Learning Rulecited by 95
- 2020Rotated Binary Neural Networkcited by 77
- 2018A Survey on Methods and Theories of Quantized Neural Networkscited by 75
- 2017Training Quantized Nets: A Deeper Understandingcited by 74
- 2017Adaptive Quantization for Deep Neural Networkcited by 67
- 2019Regularizing Activation Distribution for Training Binarized Deep Networkscited by 61
- 2017Extremely Low Bit Neural Network: Squeeze the Last Bit Out with ADMMcited by 60
- 2018ProxQuant: Quantized Neural Networks via Proximal Operatorscited by 56
- 2020A Statistical Framework for Low-bitwidth Training of Deep Neural Networkscited by 51
- 2017WRPN: Wide Reduced-Precision Networkscited by 49
- 2022AdaBin: Improving Binary Neural Networks with Adaptive Binary Setscited by 38
- 2015MuProp: Unbiased Backpropagation for Stochastic Neural Networkscited by 36
- 2020MeliusNet: Can Binary Neural Networks Achieve MobileNet-level Accuracy?cited by 32
- 2018Loss-aware Weight Quantization of Deep Networkscited by 31
- 2016Bitwise Neural Networkscited by 29
- 2021Accurate Neural Training with 4-bit Matrix Multiplications at Standard Formatscited by 26
- 2018SYQ: Learning Symmetric Quantization For Efficient Deep Neural Networkscited by 24
- 2018From Hashing to CNNs: Training BinaryWeight Networks via Hashingcited by 23
- 2017The High-Dimensional Geometry of Binary Neural Networkscited by 22
- 2020Training Binary Neural Networks using the Bayesian Learning Rulecited by 22
- 2017Minimum Energy Quantized Neural Networkscited by 20
50 most cited of 66 All 66 (JSON)
Built on by
Citing papers per year: 2023: 1; 2024: 1.
2 papers.