A Differentiable Framework for End-to-End Learning of Hybrid Structured Compression
Not on the map. Why?
Abstract (arXiv)
Filter pruning and low-rank decomposition are two of the foundational techniques for structured compression. Although recent efforts have explored hybrid approaches aiming to integrate the advantages of both techniques, their performance gains have been modest at best. In this study, we develop a textitDifferentiable Framework~(DF) that can express filter selection, rank selection, and budget constraint into a single analytical formulation. Within the framework, we introduce DML-S for filter selection, integrating scheduling into existing mask learning techniques. Additionally, we present DTL-S for rank selection, utilizing a singular value thresholding operator. The framework with DML-S and DTL-S offers a hybrid structured compression methodology that facilitates end-to-end learning through gradient-base optimization. Experimental results demonstrate the efficacy of DF, surpassing state-of-the-art structured compression methods. Our work establishes a robust and versatile avenue for advancing structured compression techniques.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 52 of its 72 references that are in the map. First posted on arXiv 21 Sep 2023.
Builds on
52 of 72 references.
References · 52
- 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
- 2016Pruning Filters for Efficient ConvNetscited by 1,143
- 2021Scaling Vision Transformerscited by 999
- 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 702
- 2017Learning Efficient Convolutional Networks through Network Slimmingcited by 676
- 2018Rethinking the Value of Network Pruningcited by 599
- 2017ThiNet: A Filter Level Pruning Method for Deep Neural Network Compressioncited by 442
- 2018AMC: AutoML for Model Compression and Acceleration on Mobile Devicescited by 378
- 2018Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Accelerationcited by 367
- 2018Soft Filter Pruning for Accelerating Deep Convolutional Neural Networkscited by 315
- 2016EIE: Efficient Inference Engine on Compressed Deep Neural Networkcited by 295
- 2020HRank: Filter Pruning using High-Rank Feature Mapcited by 292
38 more
- 2014Speeding up Convolutional Neural Networks with Low Rank Expansionscited by 284
- 2019MetaPruning: Meta Learning for Automatic Neural Network Channel Pruningcited by 227
- 2017Data-Driven Sparse Structure Selection for Deep Neural Networkscited by 181
- 2018Discrimination-aware Channel Pruning for Deep Neural Networkscited by 171
- 2015Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applicationscited by 162
- 2018Dynamic Channel Pruning: Feature Boosting and Suppressioncited by 129
- 2015Accelerating Very Deep Convolutional Networks for Classification and Detectioncited by 109
- 2015Convolutional neural networks with low-rank regularizationcited by 103
- 2019Provable Filter Pruning for Efficient Neural Networkscited by 98
- 2020SCOP: Scientific Control for Reliable Neural Network Pruningcited by 93
- 2018Channel Gating Neural Networkscited by 80
- 2021CHIP: CHannel Independence-based Pruning for Compact Neural Networkscited by 78
- 2018AutoPruner: An End-to-End Trainable Filter Pruning Method for Efficient Deep Model Inferencecited by 77
- 2019Towards Efficient Model Compression via Learned Global Rankingcited by 75
- 2020Operation-Aware Soft Channel Pruning using Differentiable Maskscited by 74
- 2020Stable Low-rank Tensor Decomposition for Compression of Convolutional Neural Networkcited by 74
- 2021Convolutional Neural Network Pruning with Structural Redundancy Reductioncited by 73
- 2019Pruning from Scratchcited by 72
- 2020Group Sparsity: The Hinge Between Filter Pruning and Decomposition for Network Compressioncited by 72
- 2020DMCP: Differentiable Markov Channel Pruning for Neural Networkscited by 69
- 2019Neural Network Pruning with Residual-Connections and Limited-Datacited by 52
- 2017Compression-aware Training of Deep Networkscited by 46
- 2021Manifold Regularized Dynamic Network Pruningcited by 43
- 2016Faster CNNs with Direct Sparse Convolutions and Guided Pruningcited by 39
- 2021Towards Compact CNNs via Collaborative Compressioncited by 36
- 2022CHEX: CHannel EXploration for CNN Model Compressioncited by 36
- 2021Compressing Neural Networks: Towards Determining the Optimal Layer-wise Decompositioncited by 33
- 2020TRP: Trained Rank Pruning for Efficient Deep Neural Networkscited by 32
- 2021Topology-Aware Network Pruning using Multi-stage Graph Embedding and Reinforcement Learningcited by 30
- 2021DECORE: Deep Compression with Reinforcement Learningcited by 29
- 2020Heuristic Rank Selection with Progressively Searching Tensor Ring Networkcited by 28
- 2018Coreset-Based Neural Network Compressioncited by 27
- 2018Asymptotic Soft Filter Pruning for Deep Convolutional Neural Networkscited by 27
- 2017Coordinating Filters for Faster Deep Neural Networkscited by 24
- 2021Fire Together Wire Together: A Dynamic Pruning Approach with Self-Supervised Mask Predictioncited by 24
- 2020Nimble: Lightweight and Parallel GPU Task Scheduling for Deep Learningcited by 17
- 2019Towards Compact ConvNets via Structure-Sparsity Regularized Filter Pruningcited by 14
- 2022Neural Network Pruning by Cooperative Coevolutioncited by 9
50 most cited of 52 All 52 (JSON)
Built on by
Citing papers per year: 2025: 1.
1 paper.