mlsanityData to 14 Oct 2026

UPSCALE: Unconstrained Channel Pruning

arXiv 2307.08771 ↗First posted 17 Jul 2023cs.CVcited by 4

Not on the map. Why?

Abstract (arXiv)

As neural networks grow in size and complexity, inference speeds decline. To combat this, one of the most effective compression techniques -- channel pruning -- removes channels from weights. However, for multi-branch segments of a model, channel removal can introduce inference-time memory copies. In turn, these copies increase inference latency -- so much so that the pruned model can be slower than the unpruned model. As a workaround, pruners conventionally constrain certain channels to be pruned together. This fully eliminates memory copies but, as we show, significantly impairs accuracy. We now have a dilemma: Remove constraints but increase latency, or add constraints and impair accuracy. In response, our insight is to reorder channels at export time, (1) reducing latency by reducing memory copies and (2) improving accuracy by removing constraints. Using this insight, we design a generic algorithm UPSCALE to prune models with any pruning pattern. By removing constraints from existing pruners, we improve ImageNet accuracy for post-training pruned models by 2.1 points on average -- benefiting DenseNet (+16.9), EfficientNetV2 (+7.9), and ResNet (+6.2). Furthermore, by reordering channels, UPSCALE improves inference speeds by up to 2x over a baseline export.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 44 of its 56 references that are in the map. First posted on arXiv 17 Jul 2023.

Builds on

44 of 56 references.

References · 44

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  3. 2016Densely Connected Convolutional Networkscited by 6,720
  4. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  5. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  6. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  7. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  8. 2019Searching for MobileNetV3cited by 1,994
  9. 2016Pruning Filters for Efficient ConvNetscited by 1,143
  10. 2021EfficientNetV2: Smaller Models and Faster Trainingcited by 1,135
  11. 2016SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model sizecited by 1,060
  12. 2018MnasNet: Platform-Aware Neural Architecture Search for Mobilecited by 943
32 more
  1. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 812
  2. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 702
  3. 2017Learning Efficient Convolutional Networks through Network Slimmingcited by 676
  4. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 603
  5. 2018Rethinking the Value of Network Pruningcited by 599
  6. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 592
  7. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 583
  8. 2020What is the State of Neural Network Pruning?cited by 583
  9. 2019The State of Sparsity in Deep Neural Networkscited by 476
  10. 2017Efficient Processing of Deep Neural Networks: A Tutorial and Surveycited by 434
  11. 2018AMC: AutoML for Model Compression and Acceleration on Mobile Devicescited by 378
  12. 2018Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Accelerationcited by 367
  13. 2020HRank: Filter Pruning using High-Rank Feature Mapcited by 292
  14. 2017Variational Dropout Sparsifies Deep Neural Networkscited by 265
  15. 2016Dynamic Network Surgery for Efficient DNNscited by 233
  16. 2015Structured Pruning of Deep Convolutional Neural Networkscited by 149
  17. 2020Layer-adaptive sparsity for the Magnitude-based Pruningcited by 126
  18. 2019Winning the Lottery with Continuous Sparsificationcited by 100
  19. 2016Training Sparse Neural Networkscited by 73
  20. 2022Revisiting Random Channel Pruning for Neural Network Compressioncited by 50
  21. 2021Recent Advances on Neural Network Pruning at Initializationcited by 41
  22. 2022The Combinatorial Brain Surgeon: Pruning Weights That Cancel One Another in Neural Networkscited by 41
  23. 2017Block-Sparse Recurrent Neural Networkscited by 39
  24. 2022SPDY: Accurate Pruning with Speedup Guaranteescited by 36
  25. 2022Pruning-as-Search: Efficient Neural Architecture Search via Channel Pruning and Structural Reparameterizationcited by 32
  26. 2021Topology-Aware Network Pruning using Multi-stage Graph Embedding and Reinforcement Learningcited by 30
  27. 2022Sparse Double Descent: Where Network Pruning Aggravates Overfittingcited by 26
  28. 2018Learning to Prune Filters in Convolutional Neural Networkscited by 25
  29. 2022Neural Network Pruning by Cooperative Coevolutioncited by 9
  30. 2018PCAS: Pruning Channels with Attention Statistics for Deep Network Compressioncited by 7
  31. 2021Training Structured Neural Networks Through Manifold Identification and Variance Reductioncited by 7
  32. 2016A work-efficient parallel sparse matrix-sparse vector multiplication algorithmcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 1; 2026: 1.

4 papers.

  1. 2024DISP-LLM: Dimension-Independent Structural Pruning for Large Language Modelscited by 24
  2. 2024SNP: Structured Neuron-level Pruning to Preserve Attention Scorescited by 5
  3. 2025Long-Tailed Data Classification by Increasing and Decreasing Neurons During Trainingcited by 0
  4. 2026Meta-Learning Guided Pruning for Few-Shot Plant Pathology on Edge Devicescited by 0