# Width Transfer: On the (In)variance of Width Optimization

arXiv: https://arxiv.org/abs/2104.13255 · HTML: https://mlsanity.com/paper/2104.13255/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 32 references that are in the map.
- First posted on arXiv 24 Apr 2021.

## Builds on (most cited first, 28 of 28)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Pruning Filters for Efficient ConvNets](https://mlsanity.com/paper/1608.08710/) (2016)
- [Learning Transferable Architectures for Scalable Image Recognition](https://mlsanity.com/paper/1707.07012/) (2017)
- [Designing Network Design Spaces](https://mlsanity.com/paper/2003.13678/) (2020)
- [Learning Efficient Convolutional Networks through Network Slimming](https://mlsanity.com/paper/1708.06519/) (2017)
- [Rethinking the Value of Network Pruning](https://mlsanity.com/paper/1810.05270/) (2018)
- [Importance Estimation for Neural Network Pruning](https://mlsanity.com/paper/1906.10771/) (2019)
- [AMC: AutoML for Model Compression and Acceleration on Mobile Devices](https://mlsanity.com/paper/1802.03494/) (2018)
- [Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks](https://mlsanity.com/paper/1808.06866/) (2018)
- [MetaPruning: Meta Learning for Automatic Neural Network Channel Pruning](https://mlsanity.com/paper/1903.10258/) (2019)
- [MorphNet: Fast & Simple Resource-Constrained Structure Learning of Deep Networks](https://mlsanity.com/paper/1711.06798/) (2017)
- [Network Pruning via Transformable Architecture Search](https://mlsanity.com/paper/1905.09717/) (2019)
- [NSGANetV2: Evolutionary Multi-Objective Surrogate-Assisted Neural Architecture Search](https://mlsanity.com/paper/2007.10396/) (2020)
- [Towards Efficient Model Compression via Learned Global Ranking](https://mlsanity.com/paper/1904.12368/) (2019)
- [DMCP: Differentiable Markov Channel Pruning for Neural Networks](https://mlsanity.com/paper/2005.03354/) (2020)
- [Regularizing Activation Distribution for Training Binarized Deep Networks](https://mlsanity.com/paper/1904.02823/) (2019)
- [EcoNAS: Finding Proxies for Economical Neural Architecture Search](https://mlsanity.com/paper/2001.01233/) (2020)
- [DHP: Differentiable Meta Pruning via HyperNetworks](https://mlsanity.com/paper/2003.13683/) (2020)
- [Probabilistic Neural Architecture Search](https://mlsanity.com/paper/1902.05116/) (2019)
- [Transfer Learning with Neural AutoML](https://mlsanity.com/paper/1803.02780/) (2018)
- [One Weight Bitwidth to Rule Them All](https://mlsanity.com/paper/2008.09916/) (2020)
- [NASTransfer: Analyzing Architecture Transferability in Large Scale Neural Architecture Search](https://mlsanity.com/paper/2006.13314/) (2020)

## Built on by (most cited first, 1 of 1)

- [Searching for Network Width with Bilaterally Coupled Network](https://mlsanity.com/paper/2203.13714/) (2022)

## Abstract (arXiv)

Optimizing the channel counts for different layers of a CNN has shown great promise in improving the efficiency of CNNs at test-time. However, these methods often introduce large computational overhead (e.g., an additional 2x FLOPs of standard training). Minimizing this overhead could therefore significantly speed up training. In this work, we propose width transfer, a technique that harnesses the assumptions that the optimized widths (or channel counts) are regular across sizes and depths. We show that width transfer works well across various width optimization algorithms and networks. Specifically, we can achieve up to 320x reduction in width optimization overhead without compromising the top-1 accuracy on ImageNet, making the additional cost of width optimization negligible relative to initial training. Our findings not only suggest an efficient way to conduct width optimization but also highlight that the widths that lead to better accuracy are invariant to various aspects of network architectures and training data.