mlsanityData to 14 Oct 2026

On Model Compression for Neural Networks: Framework, Algorithm, and Convergence Guarantee

arXiv 2303.06815 ↗First posted 13 Mar 2023cs.LG stat.MLcited by 1

Not on the map. Why?

Abstract (arXiv)

Model compression is a crucial part of deploying neural networks (NNs), especially when the memory and storage of computing devices are limited in many applications. This paper focuses on two model compression techniques: low-rank approximation and weight pruning in neural networks, which are very popular nowadays. However, training NN with low-rank approximation and weight pruning always suffers significant accuracy loss and convergence issues. In this paper, a holistic framework is proposed for model compression from a novel perspective of nonconvex optimization by designing an appropriate objective function. Then, we introduce NN-BCD, a block coordinate descent (BCD) algorithm to solve the nonconvex optimization. One advantage of our algorithm is that an efficient iteration scheme can be derived with closed-form, which is gradient-free. Therefore, our algorithm will not suffer from vanishing/exploding gradient problems. Furthermore, with the Kurdyka-Lojasiewicz (KL) property of our objective function, we show that our algorithm globally converges to a critical point at the rate of O(1/k), where k denotes the number of iterations. Lastly, extensive experiments with tensor train decomposition and weight pruning demonstrate the efficiency and superior performance of the proposed framework. Our code implementation is available at https://github.com/ChenyangLi-97/NN-BCD

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 43 of its 100 references that are in the map. First posted on arXiv 13 Mar 2023.

Builds on

43 of 100 references.

References · 43

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2021Convex optimizationcited by 2,515
  4. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  5. 2020Knowledge Distillation: A Surveycited by 1,603
  6. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 583
  7. 2016Learning Structured Sparsity in Deep Neural Networkscited by 541
  8. 2019Importance Estimation for Neural Network Pruningcited by 541
  9. 2017ThiNet: A Filter Level Pruning Method for Deep Neural Network Compressioncited by 442
  10. 2015Tensorizing Neural Networkscited by 250
  11. 2016Dynamic Network Surgery for Efficient DNNscited by 233
  12. 2015Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applicationscited by 162
31 more
  1. 2018Stochastic subgradient method converges on tame functionscited by 124
  2. 2018A Systematic DNN Weight Pruning Framework using Alternating Direction Method of Multiplierscited by 120
  3. 2016TensorLy: Tensor Learning in Pythoncited by 104
  4. 2019Winning the Lottery with Continuous Sparsificationcited by 100
  5. 2020Neural Pruning via Growing Regularizationcited by 97
  6. 2018Which Neural Net Architectures Give Rise To Exploding and Vanishing Gradients?cited by 94
  7. 2016Training Neural Networks Without Gradients: A Scalable ADMM Approachcited by 61
  8. 2017Tensor-Train Recurrent Neural Networks for Video Classificationcited by 59
  9. 2016Ultimate tensorization: compressing convolutional and FC layers alikecited by 56
  10. 2021Towards Efficient Tensor Decomposition-Based DNN Model Compression with Optimization Frameworkcited by 52
  11. 2018Global Convergence of Block Coordinate Descent in Deep Learningcited by 38
  12. 2018Compressing Recurrent Neural Networks with Tensor Ring for Action Recognitioncited by 34
  13. 2014A globally convergent algorithm for nonconvex optimization based on block coordinate updatecited by 29
  14. 2018Deep Neural Network Compression with Single and Multiple Level Quantizationcited by 27
  15. 2017Convergent Block Coordinate Descent for Training Tikhonov Regularized Deep Neural Networkscited by 26
  16. 2019Predicting Solar Flares Using a Long Short-Term Memory Networkcited by 25
  17. 2021Pruning Randomly Initialized Neural Networks with Iterative Randomizationcited by 20
  18. 2018A Proximal Block Coordinate Descent Algorithm for Deep Neural Network Trainingcited by 15
  19. 2019On ADMM in Deep Learning: Convergence and Saturation-Avoidancecited by 15
  20. 2021A Bregman Learning Framework for Sparse Neural Networkscited by 15
  21. 2021Connectivity Matters: Neural Network Pruning Through the Lens of Effective Sparsitycited by 14
  22. 2018High-dimension Tensor Completion via Gradient-based Optimization Under Tensor-train Formatcited by 12
  23. 2020Predicting Coronal Mass Ejections Using SDO/HMI Vector Magnetic Data Products and Recurrent Neural Networkscited by 11
  24. 2022Lifted Bregman Training of Neural Networkscited by 10
  25. 2017Model compression as constrained optimization, with application to neural nets. Part II: quantizationcited by 8
  26. 2021Weight Initialization Techniques for Deep Learning Algorithms in Remote Sensing: Recent Trends and Future Perspectivescited by 8
  27. 2017Model compression as constrained optimization, with application to neural nets. Part I: general frameworkcited by 6
  28. 2022Smooth Robust Tensor Completion for Background/Foreground Separation with Missing Pixels: Novel Algorithm with Convergence Guaranteecited by 5
  29. 2022A Deep Learning Approach to Generating Photospheric Vector Magnetograms of Solar Active Regions for SOHO/MDI Using SDO/HMI and BBSO Datacited by 5
  30. 2021Human Activity Recognition Using Multichannel Convolutional Neural Networkcited by 3
  31. 2021Model compression as constrained optimization, with application to neural nets. Part V: combining compressionscited by 3

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024On-Device Language Models: A Comprehensive Reviewcited by 104