Data to 14 Oct 2026

Universally Slimmable Networks and Improved Training Techniques

arXiv 1903.05134 ↗First posted 12 Mar 2019cs.CV cs.AIcited by 221

Not on the map. Why?

Abstract (arXiv)

Slimmable networks are a family of neural networks that can instantly adjust the runtime width. The width can be chosen from a predefined widths set to adaptively optimize accuracy-efficiency trade-offs at runtime. In this work, we propose a systematic approach to train universally slimmable networks (US-Nets), extending slimmable networks to execute at arbitrary width, and generalizing to networks both with and without batch normalization layers. We further propose two improved training techniques for US-Nets, named the sandwich rule and inplace distillation, to enhance training process and boost testing accuracy. We show improved performance of universally slimmable MobileNet v1 and MobileNet v2 on ImageNet classification task, compared with individually trained ones and 4-switch slimmable network baselines. We also evaluate the proposed US-Nets and improved training techniques on tasks of image super-resolution and deep reinforcement learning. Extensive ablation experiments on these representative tasks demonstrate the effectiveness of our proposed methods. Our discovery opens up the possibility to directly evaluate FLOPs-Accuracy spectrum of network architectures. Code and models are available at: https://github.com/JiahuiYu/slimmable_networks

Key facts. Cited by 221 papers in the map (data through 14 Oct 2026). First posted on arXiv 12 Mar 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 58; 2022: 42; 2023: 47; 2024: 28; 2025: 21; 2026: 25.

221 papers.

Sort:

By area:

2021AutoFormer: Searching Transformers for Visual Recognitioncited by 188
2023Structured Pruning for Deep Convolutional Neural Networks: A surveycited by 122
2022Rethinking Vision Transformers for MobileNet Size and Speedcited by 121
2023MatFormer: Nested Transformer for Elastic Inferencecited by 85
2021Dynamic Slimmable Networkcited by 83
2021Adaptive Inference through Early-Exit Networks: Design, Challenges and Directionscited by 75
2021BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Searchcited by 63
2021A Survey on Green Deep Learningcited by 52
2022Revisiting Random Channel Pruning for Neural Network Compressioncited by 50
2023EVC: Towards Real-Time Neural Image Compression with Mask Decaycited by 49
2024Flextron: Many-in-One Flexible Large Language Modelcited by 45
2021MQBench: Towards Reproducible and Deployable Model Quantization Benchmarkcited by 40

38 more

2021Slimmable Compressive Autoencoders for Practical Neural Image Compressioncited by 39
2021Teachers Do More Than Teach: Compressing Image-to-Image Modelscited by 37
2022Automated Progressive Learning for Efficient Training of Vision Transformerscited by 37
2021AlphaNet: Improved Training of Supernets with Alpha-Divergencecited by 36
2022CD2-pFed: Cyclic Distillation-guided Channel Decoupling for Model Personalization in Federated Learningcited by 34
2021Recursive-NeRF: An Efficient and Dynamically Growing NeRFcited by 32
2022Enable Deep Learning on Mobile Devices: Methods, Systems, and Applicationscited by 32
2021FedHM: Efficient Federated Learning for Heterogeneous Models via Low-rank Factorizationcited by 31
2021Accelerating BERT Inference for Sequence Labeling via Early-Exitcited by 29
2021GAIA: A Transfer Learning System of Object Detection that Fits Your Needscited by 27
2021CompOFA: Compound Once-For-All Networks for Faster Multi-Platform Deploymentcited by 26
2021ViPNAS: Efficient Video Pose Estimation via Neural Architecture Searchcited by 26
2023Stitchable Neural Networkscited by 26
2021DPICT: Deep Progressive Image Compression Using Trit-Planescited by 25
2023Cuttlefish: Low-Rank Model Training without All the Tuningcited by 25
2024AdaRevD: Adaptive Patch Exiting Reversible Decoder Pushes the Limit of Image Deblurringcited by 25
2021Towards Compact Single Image Super-Resolution via Contrastive Self-distillationcited by 24
2023Denoising Diffusion Step-aware Modelscited by 24
2022ZoomNAS: Searching for Whole-body Human Pose Estimation in the Wildcited by 23
2021Bit-Mixer: Mixed-precision networks with runtime bit-width selectioncited by 22
2023Deep Graph Reprogrammingcited by 22
2024Adapting to Length Shift: FlexiLength Network for Trajectory Predictioncited by 22
2022Contrastive Deep Supervisioncited by 20
2021Information Theoretic Representation Distillationcited by 19
2022ARM: Any-Time Super-Resolution Methodcited by 19
2023AdaptiveNet: Post-deployment Neural Architecture Adaptation for Diverse Edge Environmentscited by 19
2021Dynamic-OFA: Runtime DNN Architecture Switching for Performance Scaling on Heterogeneous Embedded Platformscited by 18
2021Masked Training of Neural Networks with Partial Gradientscited by 16
2022Adaptive Patch Exiting for Scalable Single Image Super-Resolutioncited by 16
2024Agreement-Based Cascading for Efficient Inferencecited by 16
2021Searching by Generating: Flexible and Efficient One-Shot NAS with Architecture Generatorcited by 15
2023SlimmeRF: Slimmable Radiance Fieldscited by 15
2021Dual Causal/Non-Causal Self-Attention for Streaming End-to-End Speech Recognitioncited by 14
2021Joint Superposition Coding and Training for Federated Learning over Multi-Width Neural Networkscited by 14
2021HyperSegNAS: Bridging One-Shot Neural Architecture Search with 3D Medical Image Segmentation using HyperNetcited by 14
2021NetAdaptV2: Efficient Neural Architecture Search with Fast Super-Network Training and Architecture Optimizationcited by 13
2021MutualNet: Adaptive ConvNet via Mutual Learning from Different Model Configurationscited by 13
2021Layer Pruning on Demand with Intermediate CTCcited by 13

50 most cited of 221 All 221 (JSON)