Quantized Sparse Weight Decomposition for Neural Network Compression
Not on the map. Why?
Abstract (arXiv)
In this paper, we introduce a novel method of neural network weight compression. In our method, we store weight tensors as sparse, quantized matrix factors, whose product is computed on the fly during inference to generate the target model's weights. We use projected gradient descent methods to find quantized and sparse factorization of the weight tensors. We show that this approach can be seen as a unification of weight SVD, vector quantization, and sparse PCA. Combined with end-to-end fine-tuning our method exceeds or is on par with previous state-of-the-art methods in terms of the trade-off between accuracy and model size. Our method is applicable to both moderate compression regimes, unlike vector quantization, and extreme compression regimes.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 32 of its 49 references that are in the map. First posted on arXiv 22 Jul 2022.
Builds on
32 of 49 references.
References · 32
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
- 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
- 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 697
- 2019Learned Step Size Quantizationcited by 604
- 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 596
- 2017Learning Sparse Neural Networks through L0 Regularizationcited by 571
- 2018Quantizing deep convolutional networks for efficient inference: A whitepapercited by 507
- 2019The State of Sparsity in Deep Neural Networkscited by 471
- 2021A White Paper on Neural Network Quantizationcited by 453
- 2018HAQ: Hardware-Aware Automated Quantization with Mixed Precisioncited by 426
- 2019HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precisioncited by 387
20 more
- 2014Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluationcited by 304
- 2014Speeding up Convolutional Neural Networks with Low Rank Expansionscited by 283
- 2019PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimizationcited by 234
- 2013Predicting Parameters in Deep Learningcited by 223
- 2015Quantized Convolutional Neural Networks for Mobile Devicescited by 212
- 2016Trained Ternary Quantizationcited by 203
- 2014Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decompositioncited by 199
- 2014Compressing Deep Convolutional Networks using Vector Quantizationcited by 189
- 2020LSQ+: Improving low-bit quantization through learnable offsets and better initializationcited by 180
- 2015Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applicationscited by 161
- 2017Towards Accurate Binary Convolutional Neural Networkcited by 148
- 2020Training with Quantization Noise for Extreme Model Compressioncited by 145
- 2015Accelerating Very Deep Convolutional Networks for Classification and Detectioncited by 109
- 2020Bayesian Bits: Unifying Quantization and Pruningcited by 93
- 2019And the Bit Goes Down: Revisiting the Quantization of Neural Networkscited by 78
- 2018Compressing Neural Networks using the Variational Information Bottleneckcited by 59
- 2017Structured Bayesian Pruning via Log-Normal Multiplicative Noisecited by 39
- 2020Permute, Quantize, and Fine-tune: Efficient Compression of Neural Networkscited by 28
- 2019Mixed Precision DNNs: All you need is a good parametrizationcited by 18
- 2020One-Bit Compressed Sensing via One-Shot Hard Thresholdingcited by 3
Built on by
Citing papers per year: 2023: 2; 2024: 0; 2025: 0; 2026: 1.
3 papers.