mlsanityData to 14 Oct 2026

ParaDiS: Parallelly Distributable Slimmable Neural Networks

arXiv 2110.02724 ↗First posted 6 Oct 2021cs.LG cs.AI cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

When several limited power devices are available, one of the most efficient ways to make profit of these resources, while reducing the processing latency and communication load, is to run in parallel several neural sub-networks and to fuse the result at the end of processing. However, such a combination of sub-networks must be trained specifically for each particular configuration of devices (characterized by number of devices and their capacities) which may vary over different model deployments and even within the same deployment. In this work we introduce parallelly distributable slimmable (ParaDiS) neural networks that are splittable in parallel among various device configurations without retraining. While inspired by slimmable networks allowing instant adaptation to resources on just one device, ParaDiS networks consist of several multi-device distributable configurations or switches that strongly share the parameters between them. We evaluate ParaDiS framework on MobileNet v1 and ResNet-50 architectures on ImageNet classification task and WDSR architecture for image super-resolution task. We show that ParaDiS switches achieve similar or better accuracy than the individual models, i.e., distributed models of the same structure trained individually. Moreover, we show that, as compared to universally slimmable networks that are not distributable, the accuracy of distributable ParaDiS switches either does not drop at all or drops by a maximum of 1 % only in the worst cases. Finally, once distributed over several devices, ParaDiS outperforms greatly slimmable models.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 21 of its 37 references that are in the map. First posted on arXiv 6 Oct 2021.

Builds on

21 of 37 references.

References · 21

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  4. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,845
  5. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,311
  6. 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,200
  7. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 787
  8. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 543
  9. 2018Slimmable Neural Networkscited by 305
  10. 2019Universally Slimmable Networks and Improved Training Techniquescited by 221
  11. 2020BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Modelscited by 188
  12. 2017Distributed Deep Neural Networks over the Cloud, the Edge and End Devicescited by 120
9 more
  1. 2021Split Computing and Early Exiting for Deep Learning Applications: Survey and Research Challengescited by 115
  2. 2021Dynamic Slimmable Networkcited by 83
  3. 2019Ensemble Knowledge Distillation for Learning Improved and Efficient Networkscited by 46
  4. 2021CompOFA: Compound Once-For-All Networks for Faster Multi-Platform Deploymentcited by 26
  5. 2019Memory- and Communication-Aware Model Compression for Distributed Deep Learning Inference on IoTcited by 13
  6. 2018Context-aware Deep Feature Compression for High-speed Visual Trackingcited by 8
  7. 2020Anytime Inference with Distilled Hierarchical Neural Ensemblescited by 3
  8. 2019Collaborative Execution of Deep Neural Networks on Internet of Things Devicescited by 1
  9. 2021Inplace knowledge distillation with teacher assistant for improved training of flexible deep neural networkscited by 1

Built on by

Not cited yet.