mlsanityData to 14 Oct 2026

ResIST: Layer-Wise Decomposition of ResNets for Distributed Training

arXiv 2107.00961 ↗First posted 2 Jul 2021cs.LG cs.CV cs.DCcited by 18

Not on the map. Why?

Abstract (arXiv)

We propose ResIST, a novel distributed training protocol for Residual Networks (ResNets). ResIST randomly decomposes a global ResNet into several shallow sub-ResNets that are trained independently in a distributed manner for several local iterations, before having their updates synchronized and aggregated into the global model. In the next round, new sub-ResNets are randomly generated and the process repeats until convergence. By construction, per iteration, ResIST communicates only a small portion of network parameters to each machine and never uses the full model during training. Thus, ResIST reduces the per-iteration communication, memory, and time requirements of ResNet training to only a fraction of the requirements of full-model training. In comparison to common protocols, like data-parallel training and data-parallel training with local SGD, ResIST yields a decrease in communication and compute requirements, while being competitive with respect to model performance.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Builds on 61 of its 90 references that are in the map. First posted on arXiv 2 Jul 2021.

Builds on

61 of 90 references.

References · 61

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  7. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  8. 2017Mask R-CNNcited by 7,277
  9. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  10. 2016Densely Connected Convolutional Networkscited by 6,720
  11. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  12. 2014Going Deeper with Convolutionscited by 4,241
38 more
  1. 2018Neural Ordinary Differential Equationscited by 3,779
  2. 2016Wide Residual Networkscited by 3,070
  3. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,546
  4. 2013Visualizing and Understanding Convolutional Networkscited by 2,290
  5. 2016Identity Mappings in Deep Residual Networkscited by 1,965
  6. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  7. 2016Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learningcited by 1,512
  8. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  9. 2016YOLO9000: Better, Faster, Strongercited by 1,431
  10. 2016Pruning Filters for Efficient ConvNetscited by 1,143
  11. 2016Deep Networks with Stochastic Depthcited by 979
  12. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  13. 2018Local SGD Converges Fast and Communicates Littlecited by 605
  14. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 574
  15. 2017Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descentcited by 502
  16. 2019Augmented Neural ODEscited by 397
  17. 2020A Unified Theory of Decentralized SGD with Changing Topology and Local Updatescited by 367
  18. 2018Horovod: fast and easy distributed deep learning in TensorFlowcited by 279
  19. 2020ReZero is All You Need: Fast Convergence at Large Depthcited by 264
  20. 2017Sparse Communication for Distributed Gradient Descentcited by 249
  21. 2019Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communicationcited by 221
  22. 2017Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equationscited by 203
  23. 2018Don't Use Large Mini-Batches, Use Local SGDcited by 202
  24. 2017Gradient Sparsification for Communication-Efficient Distributed Optimizationcited by 201
  25. 2018Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysiscited by 156
  26. 2016QSGD: Communication-Efficient SGD via Gradient Quantization and Encodingcited by 147
  27. 2019Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of Viewcited by 113
  28. 2014Deep learning with Elastic Averaging SGDcited by 90
  29. 2018Communication Compression for Decentralized Trainingcited by 88
  30. 2018Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutescited by 63
  31. 2020A Mean-field Analysis of Deep ResNet and Beyond: Towards Provable Optimization Via Overparameterization From Depthcited by 56
  32. 2020Advances in Asynchronous Parallel and Distributed Optimizationcited by 40
  33. 2020Adaptive Federated Dropout: Improving Communication Efficiency and Generalization for Federated Learningcited by 38
  34. 2018Layer-Parallel Training of Deep Residual Neural Networkscited by 35
  35. 2016Parallel SGD: When does averaging help?cited by 32
  36. 2018Efficient and Robust Parallel DNN Training through Model Parallelism on Multi-GPU Platformcited by 28
  37. 2017Extremely Large Minibatch SGD: Training ResNet-50 on ImageNet in 15 Minutescited by 22
  38. 2018High-Accuracy Low-Precision Trainingcited by 20

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 2; 2023: 7; 2024: 6; 2025: 2.

18 papers.

Sort:

By area:

  1. 2023On Efficient Training of Large-Scale Deep Learning Models: A Literature Reviewcited by 26
  2. 2024FedP3: Federated Personalized and Privacy-friendly Network Pruning under Model Heterogeneitycited by 13
  3. 2022Efficient and Light-Weight Federated Learning via Asynchronous Distributed Dropoutcited by 12
  4. 2021On the Convergence of Shallow Neural Network Training with Randomly Masked Neuronscited by 11
  5. 2024Efficient Stagewise Pretraining via Progressive Subnetworkscited by 10
  6. 2024WASH: Train your Ensemble with Communication-Efficient Weight Shuffling, then Averagecited by 9
  7. 2025Leveraging Randomness in Model and Data Partitioning for Privacy Amplificationcited by 7
  8. 2023Towards a Better Theoretical Understanding of Independent Subnetwork Trainingcited by 6
  9. 2023Federated Learning Over Images: Vertical Decompositions and Pre-Trained Backbones Are Difficult to Beatcited by 4
  10. 2023Module-wise Training of Neural Networks via the Minimizing Movement Schemecited by 4
  11. 2024F3OCUS -- Federated Finetuning of Vision-Language Foundation Models with Optimal Client Layer Updating Strategy via Multi-objective Meta-Heuristicscited by 4
  12. 2024Embracing Federated Learning: Enabling Weak Client Participation via Partial Model Trainingcited by 3
6 more
  1. 2024Exploring Selective Layer Fine-Tuning in Federated Learningcited by 3
  2. 2023Understanding Progressive Training Through the Framework of Randomized Coordinate Descentcited by 2
  3. 2022Block-wise Training of Residual Networks via the Minimizing Movement Schemecited by 1
  4. 2023Towards Hyperparameter-Agnostic DNN Training via Dynamical System Insightscited by 1
  5. 2023Federated Learning over Hierarchical Wireless Networks: Training Latency Minimization via Submodel Partitioningcited by 1
  6. 2025TwIST: Rigging the Lottery in Transformers with Independent Subnetwork Trainingcited by 1