mlsanityData to 14 Oct 2026

Model compression as constrained optimization, with application to neural nets. Part V: combining compressions

arXiv 2107.04380 ↗First posted 9 Jul 2021cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

Model compression is generally performed by using quantization, low-rank approximation or pruning, for which various algorithms have been researched in recent years. One fundamental question is: what types of compression work better for a given model? Or even better: can we improve by combining compressions in a suitable way? We formulate this generally as a problem of optimizing the loss but where the weights are constrained to equal an additive combination of separately compressed parts; and we give an algorithm to learn the corresponding parts' parameters. Experimentally with deep neural nets, we observe that 1) we can find significantly better models in the error-compression space, indicating that different compression types have complementary benefits, and 2) the best type of combination depends exquisitely on the type of neural net. For example, we can compress ResNets and AlexNet using only 1 bit per weight without error degradation at the cost of adding a few floating point weights. However, VGG nets can be better compressed by combining low-rank with a few floating point weights.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 33 of its 59 references that are in the map. First posted on arXiv 9 Jul 2021.

Builds on

33 of 59 references.

References · 33

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  4. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  5. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  6. 2016Pruning Filters for Efficient ConvNetscited by 1,143
  7. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 849
  8. 2018Rethinking the Value of Network Pruningcited by 599
  9. 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 569
  10. 2016Learning Structured Sparsity in Deep Neural Networkscited by 541
  11. 2014Caffe: Convolutional Architecture for Fast Feature Embeddingcited by 288
  12. 2015Tensorizing Neural Networkscited by 250
21 more
  1. 2017NISP: Pruning Networks using Neuron Importance Score Propagationcited by 228
  2. 2015Coordinate Descent Algorithmscited by 219
  3. 2017Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weightscited by 217
  4. 2015Quantized Convolutional Neural Networks for Mobile Devicescited by 215
  5. 2016Trained Ternary Quantizationcited by 208
  6. 2016Ternary Weight Networkscited by 204
  7. 2019Quantization Networkscited by 153
  8. 2017Soft-to-Hard Vector Quantization for End-to-End Learning Compressible Representationscited by 152
  9. 2018Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layerscited by 112
  10. 2017Extremely Low Bit Neural Network: Squeeze the Last Bit Out with ADMMcited by 61
  11. 2016Towards the Limit of Network Quantizationcited by 50
  12. 2017Compression-aware Training of Deep Networkscited by 46
  13. 2019Approximated Oracle Filter Pruning for Destructive CNN Width Optimizationcited by 35
  14. 2020TRP: Trained Rank Pruning for Efficient Deep Neural Networkscited by 32
  15. 2019Adaptive Loss-aware Quantization for Multi-bit Networkscited by 30
  16. 2018Deep Neural Network Compression with Single and Multiple Level Quantizationcited by 27
  17. 2017Coordinating Filters for Faster Deep Neural Networkscited by 24
  18. 2018BinaryRelax: A Relaxation Approach For Training Deep Neural Networks With Quantized Weightscited by 24
  19. 2019OICSR: Out-In-Channel Sparsity Regularization for Compact Deep Neural Networkscited by 18
  20. 2017Model compression as constrained optimization, with application to neural nets. Part I: general frameworkcited by 6
  21. 2020A flexible, extensible software framework for model compression based on the LC algorithmcited by 2

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 1.

3 papers.

  1. 2023Poster: Self-Supervised Quantization-Aware Knowledge Distillationcited by 11
  2. 2023On Model Compression for Neural Networks: Framework, Algorithm, and Convergence Guaranteecited by 1
  3. 2024Self-Supervised Quantization-Aware Knowledge Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Hybrid Compression: Integrating Pruning and Quantization for Optimized Neural Networks
  2. 2024"Lossless" Compression of Deep Neural Networks: A High-dimensional Neural Tangent Kernel Approach
  3. 2023End-to-End Neural Network Compression via fracℓ1ℓ2 Regularized Latency Surrogates
  4. 2022Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning
  5. 2026Big2Small: A Unifying Neural Network Framework for Model Compression
  6. 2025Extreme Model Compression with Structured Sparsity at Low Precision
  7. 2025Efficient compression of neural networks and datasets
  8. 2023Towards Optimal Compression: Joint Pruning and Quantization