# Universally Slimmable Networks and Improved Training Techniques

arXiv: https://arxiv.org/abs/1903.05134 · HTML: https://mlsanity.com/paper/1903.05134/

## Key facts

- Cited by 221 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Mar 2019.

## Built on by (most cited first, 30 of 221)

- [AutoFormer: Searching Transformers for Visual Recognition](https://mlsanity.com/paper/2107.00651/) (2021)
- [Structured Pruning for Deep Convolutional Neural Networks: A survey](https://mlsanity.com/paper/2303.00566/) (2023)
- [Rethinking Vision Transformers for MobileNet Size and Speed](https://mlsanity.com/paper/2212.08059/) (2022)
- [MatFormer: Nested Transformer for Elastic Inference](https://mlsanity.com/paper/2310.07707/) (2023)
- [Dynamic Slimmable Network](https://mlsanity.com/paper/2103.13258/) (2021)
- [Adaptive Inference through Early-Exit Networks: Design, Challenges and Directions](https://mlsanity.com/paper/2106.05022/) (2021)
- [BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Search](https://mlsanity.com/paper/2103.12424/) (2021)
- [A Survey on Green Deep Learning](https://mlsanity.com/paper/2111.05193/) (2021)
- [Revisiting Random Channel Pruning for Neural Network Compression](https://mlsanity.com/paper/2205.05676/) (2022)
- [EVC: Towards Real-Time Neural Image Compression with Mask Decay](https://mlsanity.com/paper/2302.05071/) (2023)
- [Flextron: Many-in-One Flexible Large Language Model](https://mlsanity.com/paper/2406.10260/) (2024)
- [MQBench: Towards Reproducible and Deployable Model Quantization Benchmark](https://mlsanity.com/paper/2111.03759/) (2021)
- [Slimmable Compressive Autoencoders for Practical Neural Image Compression](https://mlsanity.com/paper/2103.15726/) (2021)
- [Teachers Do More Than Teach: Compressing Image-to-Image Models](https://mlsanity.com/paper/2103.03467/) (2021)
- [Automated Progressive Learning for Efficient Training of Vision Transformers](https://mlsanity.com/paper/2203.14509/) (2022)
- [AlphaNet: Improved Training of Supernets with Alpha-Divergence](https://mlsanity.com/paper/2102.07954/) (2021)
- [CD2-pFed: Cyclic Distillation-guided Channel Decoupling for Model Personalization in Federated Learning](https://mlsanity.com/paper/2204.03880/) (2022)
- [Recursive-NeRF: An Efficient and Dynamically Growing NeRF](https://mlsanity.com/paper/2105.09103/) (2021)
- [Enable Deep Learning on Mobile Devices: Methods, Systems, and Applications](https://mlsanity.com/paper/2204.11786/) (2022)
- [FedHM: Efficient Federated Learning for Heterogeneous Models via Low-rank Factorization](https://mlsanity.com/paper/2111.14655/) (2021)
- [Accelerating BERT Inference for Sequence Labeling via Early-Exit](https://mlsanity.com/paper/2105.13878/) (2021)
- [GAIA: A Transfer Learning System of Object Detection that Fits Your Needs](https://mlsanity.com/paper/2106.11346/) (2021)
- [CompOFA: Compound Once-For-All Networks for Faster Multi-Platform Deployment](https://mlsanity.com/paper/2104.12642/) (2021)
- [ViPNAS: Efficient Video Pose Estimation via Neural Architecture Search](https://mlsanity.com/paper/2105.10154/) (2021)
- [Stitchable Neural Networks](https://mlsanity.com/paper/2302.06586/) (2023)
- [DPICT: Deep Progressive Image Compression Using Trit-Planes](https://mlsanity.com/paper/2112.06334/) (2021)
- [Cuttlefish: Low-Rank Model Training without All the Tuning](https://mlsanity.com/paper/2305.02538/) (2023)
- [AdaRevD: Adaptive Patch Exiting Reversible Decoder Pushes the Limit of Image Deblurring](https://mlsanity.com/paper/2406.09135/) (2024)
- [Towards Compact Single Image Super-Resolution via Contrastive Self-distillation](https://mlsanity.com/paper/2105.11683/) (2021)
- [Denoising Diffusion Step-aware Models](https://mlsanity.com/paper/2310.03337/) (2023)

## Abstract (arXiv)

Slimmable networks are a family of neural networks that can instantly adjust the runtime width. The width can be chosen from a predefined widths set to adaptively optimize accuracy-efficiency trade-offs at runtime. In this work, we propose a systematic approach to train universally slimmable networks (US-Nets), extending slimmable networks to execute at arbitrary width, and generalizing to networks both with and without batch normalization layers. We further propose two improved training techniques for US-Nets, named the sandwich rule and inplace distillation, to enhance training process and boost testing accuracy. We show improved performance of universally slimmable MobileNet v1 and MobileNet v2 on ImageNet classification task, compared with individually trained ones and 4-switch slimmable network baselines. We also evaluate the proposed US-Nets and improved training techniques on tasks of image super-resolution and deep reinforcement learning. Extensive ablation experiments on these representative tasks demonstrate the effectiveness of our proposed methods. Our discovery opens up the possibility to directly evaluate FLOPs-Accuracy spectrum of network architectures. Code and models are available at: https://github.com/JiahuiYu/slimmable_networks