# Locally Free Weight Sharing for Network Width Search

arXiv: https://arxiv.org/abs/2102.05258 · HTML: https://mlsanity.com/paper/2102.05258/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 59 references that are in the map.
- First posted on arXiv 10 Feb 2021.

## Builds on (most cited first, 30 of 40)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Pruning Filters for Efficient ConvNets](https://mlsanity.com/paper/1608.08710/) (2016)
- [Multi-Task Learning as Multi-Objective Optimization](https://mlsanity.com/paper/1810.04650/) (2018)
- [Once-for-All: Train One Network and Specialize it for Efficient Deployment](https://mlsanity.com/paper/1908.09791/) (2019)
- [Channel Pruning for Accelerating Very Deep Neural Networks](https://mlsanity.com/paper/1707.06168/) (2017)
- [Learning Efficient Convolutional Networks through Network Slimming](https://mlsanity.com/paper/1708.06519/) (2017)
- [Rethinking the Value of Network Pruning](https://mlsanity.com/paper/1810.05270/) (2018)
- [AMC: AutoML for Model Compression and Acceleration on Mobile Devices](https://mlsanity.com/paper/1802.03494/) (2018)
- [Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Acceleration](https://mlsanity.com/paper/1811.00250/) (2018)
- [Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks](https://mlsanity.com/paper/1808.06866/) (2018)
- [MetaPruning: Meta Learning for Automatic Neural Network Channel Pruning](https://mlsanity.com/paper/1903.10258/) (2019)
- [Discrimination-aware Channel Pruning for Deep Neural Networks](https://mlsanity.com/paper/1810.11809/) (2018)
- [Towards Optimal Structured CNN Pruning via Generative Adversarial Learning](https://mlsanity.com/paper/1903.09291/) (2019)
- [Dynamic Channel Pruning: Feature Boosting and Suppression](https://mlsanity.com/paper/1810.05331/) (2018)
- [Gate Decorator: Global Filter Pruning Method for Accelerating Deep Convolutional Neural Networks](https://mlsanity.com/paper/1909.08174/) (2019)
- [Channel Gating Neural Networks](https://mlsanity.com/paper/1805.12549/) (2018)
- [Network Pruning via Transformable Architecture Search](https://mlsanity.com/paper/1905.09717/) (2019)
- [AutoPruner: An End-to-End Trainable Filter Pruning Method for Efficient Deep Model Inference](https://mlsanity.com/paper/1805.08941/) (2018)
- [Pruning from Scratch](https://mlsanity.com/paper/1909.12579/) (2019)
- [Good Subnetworks Provably Exist: Pruning via Greedy Forward Selection](https://mlsanity.com/paper/2003.01794/) (2020)

## Built on by (most cited first, 27 of 27)

- [Patch Slimming for Efficient Vision Transformers](https://mlsanity.com/paper/2106.02852/) (2021)
- [Manifold Regularized Dynamic Network Pruning](https://mlsanity.com/paper/2103.05861/) (2021)
- [Dynamic Sparse Training with Structured Sparsity](https://mlsanity.com/paper/2305.02299/) (2023)
- [CHEX: CHannel EXploration for CNN Model Compression](https://mlsanity.com/paper/2203.15794/) (2022)
- [Prioritized Architecture Sampling with Monto-Carlo Tree Search](https://mlsanity.com/paper/2103.11922/) (2021)
- [ViTAS: Vision Transformer Architecture Search](https://mlsanity.com/paper/2106.13700/) (2021)
- [SOSP: Efficiently Capturing Global Correlations by Second-Order Structured Pruning](https://mlsanity.com/paper/2110.11395/) (2021)
- [BCNet: Searching for Network Width with Bilaterally Coupled Network](https://mlsanity.com/paper/2105.10533/) (2021)
- [K-shot NAS: Learnable Weight-Sharing for NAS with K-shot Supernets](https://mlsanity.com/paper/2106.06442/) (2021)
- [On Redundancy and Diversity in Cell-based Neural Architecture Search](https://mlsanity.com/paper/2203.08887/) (2022)
- [Dynamic Sparsity Is Channel-Level Sparsity Learner](https://mlsanity.com/paper/2305.19454/) (2023)
- [Trainability Preserving Neural Pruning](https://mlsanity.com/paper/2207.12534/) (2022)
- [Re-mine, Learn and Reason: Exploring the Cross-modal Semantic Correlations for Language-guided HOI detection](https://mlsanity.com/paper/2307.13529/) (2023)
- [GreedyNASv2: Greedier Search with a Greedy Path Filter](https://mlsanity.com/paper/2111.12609/) (2021)
- [DyRep: Bootstrapping Training with Dynamic Re-parameterization](https://mlsanity.com/paper/2203.12868/) (2022)
- [HRel: Filter Pruning based on High Relevance between Activation Maps and Class Labels](https://mlsanity.com/paper/2202.10716/) (2022)
- [Maxwell's Demon at Work: Efficient Pruning by Leveraging Saturation of Neurons](https://mlsanity.com/paper/2403.07688/) (2024)
- [Ensemble Knowledge Guided Sub-network Search and Fine-tuning for Filter Pruning](https://mlsanity.com/paper/2203.02651/) (2022)
- [Searching for Network Width with Bilaterally Coupled Network](https://mlsanity.com/paper/2203.13714/) (2022)
- [ScaleNet: Searching for the Model to Scale](https://mlsanity.com/paper/2207.07267/) (2022)
- [REPrune: Channel Pruning via Kernel Representative Selection](https://mlsanity.com/paper/2402.17862/) (2024)
- [Adaptive Width Neural Networks](https://mlsanity.com/paper/2501.15889/) (2025)
- [Relational Surrogate Loss Learning](https://mlsanity.com/paper/2202.13197/) (2022)
- [SUBP: Soft Uniform Block Pruning for 1xN Sparse CNNs Multithreading Acceleration](https://mlsanity.com/paper/2310.06218/) (2023)
- [Multi-scale alignment and Spatial ROI Module for COVID-19 Diagnosis](https://mlsanity.com/paper/2207.01345/) (2022)
- [Propheter: Prophetic Teacher Guided Long-Tailed Distribution Learning](https://mlsanity.com/paper/2304.04135/) (2023)
- [CR-SFP: Learning Consistent Representation for Soft Filter Pruning](https://mlsanity.com/paper/2312.11555/) (2023)

## Abstract (arXiv)

Searching for network width is an effective way to slim deep neural networks with hardware budgets. With this aim, a one-shot supernet is usually leveraged as a performance evaluator to rank the performance wrt~different width. Nevertheless, current methods mainly follow a manually fixed weight sharing pattern, which is limited to distinguish the performance gap of different width. In this paper, to better evaluate each width, we propose a locally free weight sharing strategy (CafeNet) accordingly. In CafeNet, weights are more freely shared, and each width is jointly indicated by its base channels and free channels, where free channels are supposed to loCAte FrEely in a local zone to better represent each width. Besides, we propose to further reduce the search space by leveraging our introduced FLOPs-sensitive bins. As a result, our CafeNet can be trained stochastically and get optimized within a min-min strategy. Extensive experiments on ImageNet, CIFAR-10, CelebA and MS COCO dataset have verified our superiority comparing to other state-of-the-art baselines. For example, our method can further boost the benchmark NAS network EfficientNet-B0 by 0.41% via searching its width more delicately.