mlsanityData to 7 Oct 2026

Compressing Neural Networks: Towards Determining the Optimal Layer-wise Decomposition

arXiv 2107.11442 ↗First posted 23 Jul 2021cs.LG cs.AI cs.CVcited by 33

Not on the map. Why?

Abstract (arXiv)

We present a novel global compression framework for deep neural networks that automatically analyzes each layer to identify the optimal per-layer compression ratio, while simultaneously achieving the desired overall compression. Our algorithm hinges on the idea of compressing each convolutional (or fully-connected) layer by slicing its channels into multiple groups and decomposing each group via low-rank decomposition. At the core of our algorithm is the derivation of layer-wise error bounds from the Eckart Young Mirsky theorem. We then leverage these bounds to frame the compression problem as an optimization problem where we wish to minimize the maximum compression error across layers and propose an efficient algorithm towards a solution. Our experiments indicate that our method outperforms existing low-rank compression approaches across a wide range of networks and data sets. We believe that our results open up new avenues for future research into the global performance-size trade-offs of modern neural networks. Our code is available at https://github.com/lucaslie/torchprune.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Builds on 63 of its 110 references that are in the map. First posted on arXiv 23 Jul 2021.

Builds on

63 of 110 references.

References · 63

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  5. 2016Densely Connected Convolutional Networkscited by 6,691
  6. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  7. 2016Wide Residual Networkscited by 3,057
  8. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  9. 2017Rethinking Atrous Convolution for Semantic Image Segmentationcited by 2,154
  10. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  11. 2016Pruning Filters for Efficient ConvNetscited by 1,129
  12. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 697
38 more
  1. 2017Learning Efficient Convolutional Networks through Network Slimmingcited by 673
  2. 2018Rethinking the Value of Network Pruningcited by 595
  3. 2019Importance Estimation for Neural Network Pruningcited by 531
  4. 2018Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Accelerationcited by 365
  5. 2018Soft Filter Pruning for Accelerating Deep Convolutional Neural Networkscited by 313
  6. 2018Stronger generalization bounds for deep nets via a compression approachcited by 306
  7. 2014Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluationcited by 304
  8. 2020HRank: Filter Pruning using High-Rank Feature Mapcited by 291
  9. 2014Speeding up Convolutional Neural Networks with Low Rank Expansionscited by 283
  10. 2015Tensorizing Neural Networkscited by 246
  11. 2017NISP: Pruning Networks using Neuron Importance Score Propagationcited by 228
  12. 2019MetaPruning: Meta Learning for Automatic Neural Network Channel Pruningcited by 226
  13. 2013Predicting Parameters in Deep Learningcited by 223
  14. 2015Quantized Convolutional Neural Networks for Mobile Devicescited by 212
  15. 2014Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decompositioncited by 199
  16. 2018Discrimination-aware Channel Pruning for Deep Neural Networkscited by 171
  17. 2015Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applicationscited by 161
  18. 2015Compressing Neural Networks with the Hashing Trickcited by 150
  19. 2018Dynamic Channel Pruning: Feature Boosting and Suppressioncited by 129
  20. 2020Dynamic Model Pruning with Feedbackcited by 125
  21. 2018Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layerscited by 112
  22. 2015Accelerating Very Deep Convolutional Networks for Classification and Detectioncited by 109
  23. 2015Convolutional neural networks with low-rank regularizationcited by 102
  24. 2017Soft Weight-Sharing for Neural Network Compressioncited by 99
  25. 2019Provable Filter Pruning for Efficient Neural Networkscited by 98
  26. 2020Neural Pruning via Growing Regularizationcited by 97
  27. 2018Channel Gating Neural Networkscited by 80
  28. 2018AutoPruner: An End-to-End Trainable Filter Pruning Method for Efficient Deep Model Inferencecited by 77
  29. 2019Towards Efficient Model Compression via Learned Global Rankingcited by 75
  30. 2017More is Less: A More Complicated Network with Less Inference Complexitycited by 66
  31. 2021Lost in Pruning: The Effects of Pruning Neural Networks beyond Test Accuracycited by 62
  32. 2019Learning Filter Basis for Convolutional Neural Network Compressioncited by 57
  33. 2016Ultimate tensorization: compressing convolutional and FC layers alikecited by 55
  34. 2017Compression-aware Training of Deep Networkscited by 46
  35. 2018Data-Dependent Coresets for Compressing Neural Networks with Applications to Generalization Boundscited by 43
  36. 2019Approximated Oracle Filter Pruning for Destructive CNN Width Optimizationcited by 35
  37. 2016Deep Roots: Improving CNN Efficiency with Hierarchical Filter Groupscited by 32
  38. 2018GroupReduce: Block-Wise Low-Rank Approximation for Neural Language Model Shrinkingcited by 32

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 9; 2023: 11; 2024: 7; 2025: 3; 2026: 2.

33 papers.

Sort:

  1. 2023Structured Pruning for Deep Convolutional Neural Networks: A surveycited by 122
  2. 2022Compute-Efficient Deep Learning: Algorithmic Trends and Opportunitiescited by 38
  3. 2022SPDY: Accurate Pruning with Speedup Guaranteescited by 34
  4. 2023Follow Anything: Open-set detection, tracking, and following in real-timecited by 28
  5. 2022Pruning Neural Networks via Coresets and Convex Geometry: Towards No Assumptionscited by 20
  6. 2023HALOC: Hardware-Aware Automatic Low-Rank Compression for Compact Neural Networkscited by 20
  7. 2022Compression-aware Training of Neural Networks using Frank-Wolfecited by 16
  8. 2023COMCAT: Towards Efficient Compression and Customization of Attention-Based Vision Modelscited by 14
  9. 2024ELRT: Efficient Low-Rank Training for Compact Convolutional Neural Networkscited by 11
  10. 2021Model Preserving Compression for Neural Networkscited by 9
  11. 2022Boundary between noise and information applied to filtering neural network weight matricescited by 9
  12. 2022Coresets for Data Discretization and Sine Wave Fittingcited by 6
21 more
  1. 2025LaX: Boosting Low-Rank Training of Foundation Models via Latent Crossingcited by 6
  2. 2023Low Rank Optimization for Efficient Deep Learning: Making A Balance between Compact Architecture and Fast Trainingcited by 5
  3. 2022Deep Learning on Home Drone: Searching for the Optimal Architecturecited by 4
  4. 2023Lossy and Lossless (L2) Post-training Model Size Compressioncited by 4
  5. 2023Differentiable Learning of Generalized Structured Matrices for Efficient Deep Neural Networkscited by 4
  6. 2024Revealing the Utilized Rank of Subspaces of Learning in Neural Networkscited by 4
  7. 2022SVD-NAS: Coupling Low-Rank Approximation and Neural Architecture Searchcited by 3
  8. 2023Singular Value Representation: A New Graph Perspective On Neural Networkscited by 3
  9. 2024Structure-Preserving Network Compression Via Low-Rank Induced Training Through Linear Layers Compositioncited by 3
  10. 2023How Informative is the Approximation Error from Tensor Decomposition for Neural Network Compression?cited by 2
  11. 2025BALF: Budgeted Activation-Aware Low-Rank Factorization for Fine-Tuning-Free Model Compressioncited by 2
  12. 2023A Differentiable Framework for End-to-End Learning of Hybrid Structured Compressioncited by 1
  13. 2024An Adaptive Tensor-Train Decomposition Approach for Efficient Deep Neural Network Compressioncited by 1
  14. 2025Compress to Impress: Efficient LLM Adaptation Using a Single Gradient Step on 100 Samplescited by 1
  15. 2026SLORR: Simple and Efficient In-Training Low-Rank Regularizationcited by 1
  16. 2022SeKron: A Decomposition Method Supporting Many Factorization Structurescited by 0
  17. 2023DietCNN: Multiplication-free Inference for Quantized CNNscited by 0
  18. 2024AB-Training: A Communication-Efficient Approach for Distributed Low-Rank Learningcited by 0
  19. 2024Reduced storage direct tensor ring decomposition for convolutional neural networks compressioncited by 0
  20. 2024Unified Framework for Pre-trained Neural Network Compression via Decomposition and Optimized Rank Selectioncited by 0
  21. 2026Hierarchical Spatio-Channel Clustering for Efficient Model Compression in Medical Image Analysiscited by 0