mlsanityData to 14 Oct 2026

A Highly Effective Low-Rank Compression of Deep Neural Networks with Modified Beam-Search and Modified Stable Rank

arXiv 2111.15179 ↗First posted 30 Nov 2021cs.LG cs.AI cs.CVcited by 1

Not on the map. Why?

Abstract (arXiv)

Compression has emerged as one of the essential deep learning research topics, especially for the edge devices that have limited computation power and storage capacity. Among the main compression techniques, low-rank compression via matrix factorization has been known to have two problems. First, an extensive tuning is required. Second, the resulting compression performance is typically not impressive. In this work, we propose a low-rank compression method that utilizes a modified beam-search for an automatic rank selection and a modified stable rank for a compression-friendly training. The resulting BSR (Beam-search and Stable Rank) algorithm requires only a single hyperparameter to be tuned for the desired compression ratio. The performance of BSR in terms of accuracy and compression ratio trade-off curve turns out to be superior to the previously known low-rank compression methods. Furthermore, BSR can perform on par with or better than the state-of-the-art structured pruning methods. As with pruning, BSR can be easily combined with quantization for an additional compression.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 39 of its 58 references that are in the map. First posted on arXiv 30 Nov 2021.

Builds on

39 of 58 references.

References · 39

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  4. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  5. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  6. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,077
  7. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 849
  8. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 702
  9. 2018Rethinking the Value of Network Pruningcited by 599
  10. 2017ThiNet: A Filter Level Pruning Method for Deep Neural Network Compressioncited by 442
  11. 2018AMC: AutoML for Model Compression and Acceleration on Mobile Devicescited by 378
  12. 2015Deep Learning with Limited Numerical Precisioncited by 376
27 more
  1. 2014Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluationcited by 306
  2. 2017A Survey of Model Compression and Acceleration for Deep Neural Networkscited by 301
  3. 2014Speeding up Convolutional Neural Networks with Low Rank Expansionscited by 284
  4. 2018Paraphrasing Complex Network: Network Compression via Factor Transfercited by 260
  5. 2015Quantized Convolutional Neural Networks for Mobile Devicescited by 215
  6. 2018Discrimination-aware Channel Pruning for Deep Neural Networkscited by 171
  7. 2015Fixed Point Quantization of Deep Convolutional Networkscited by 143
  8. 2020If beam search is the answer, what was the question?cited by 116
  9. 2015Data-free parameter pruning for Deep Neural Networkscited by 112
  10. 2018Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layerscited by 112
  11. 2015Accelerating Very Deep Convolutional Networks for Classification and Detectioncited by 109
  12. 2016Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1cited by 105
  13. 2015Convolutional neural networks with low-rank regularizationcited by 103
  14. 2015Fast ConvNets Using Group-wise Brain Damagecited by 82
  15. 2021CHIP: CHannel Independence-based Pruning for Compact Neural Networkscited by 78
  16. 2019Towards Efficient Model Compression via Learned Global Rankingcited by 75
  17. 2019Pruning from Scratchcited by 72
  18. 2017Compression-aware Training of Deep Networkscited by 46
  19. 2019Stable Rank Normalization for Improved Generalization in Neural Networks and GANscited by 41
  20. 2021BSQ: Exploring Bit-Level Sparsity for Mixed-Precision Neural Network Quantizationcited by 39
  21. 2017Coordinating Filters for Faster Deep Neural Networkscited by 24
  22. 2018Trained Rank Pruning for Efficient Deep Neural Networkscited by 23
  23. 2021Bit-Mixer: Mixed-precision networks with runtime bit-width selectioncited by 22
  24. 2016Randomized Matrix Decompositions using Rcited by 13
  25. 2017Domain-adaptive deep network compressioncited by 9
  26. 2018Efficient Structured Pruning and Architecture Searching for Group Convolutioncited by 4
  27. 2021Generating Natural Language Adversarial Examples through An Improved Beam Search Algorithmcited by 3

Built on by

Cited by year

Citing papers per year: 2023: 1.

1 paper.

  1. 2023How Informative is the Approximation Error from Tensor Decomposition for Neural Network Compression?cited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2023Low Rank Optimization for Efficient Deep Learning: Making A Balance between Compact Architecture and Fast Training
  2. 2024"Lossless" Compression of Deep Neural Networks: A High-dimensional Neural Tangent Kernel Approach
  3. 2023Training Acceleration of Low-Rank Decomposed Networks using Sequential Freezing and Rank Quantization
  4. 2026Hybrid Compression: Integrating Pruning and Quantization for Optimized Neural Networks
  5. 2023Neural Network Compression using Binarization and Few Full-Precision Weights
  6. 2022Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning
  7. 2023MST-compression: Compressing and Accelerating Binary Neural Networks with Minimum Spanning Tree
  8. 2021Compact representations of convolutional neural networks via weight pruning and quantization