mlsanityData to 7 Oct 2026

Towards a Better Theoretical Understanding of Independent Subnetwork Training

arXiv 2306.16484 ↗First posted 28 Jun 2023cs.LG cs.DC math.OCcited by 6

Not on the map. Why?

Abstract (arXiv)

Modern advancements in large-scale machine learning would be impossible without the paradigm of data-parallel distributed computing. Since distributed computing with large-scale models imparts excessive pressure on communication channels, significant recent research has been directed toward co-designing communication compression strategies and training algorithms with the goal of reducing communication costs. While pure data parallelism allows better data scaling, it suffers from poor model scaling properties. Indeed, compute nodes are severely limited by memory constraints, preventing further increases in model size. For this reason, the latest achievements in training giant neural network models also rely on some form of model parallelism. In this work, we take a closer theoretical look at Independent Subnetwork Training (IST), which is a recently proposed and highly effective technique for solving the aforementioned problems. We identify fundamental differences between IST and alternative approaches, such as distributed methods with compressed communication, and provide a precise analysis of its optimization performance on a quadratic model.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 187 of its 256 references that are in the map. First posted on arXiv 28 Jun 2023.

Builds on

187 of 256 references.

References · 187

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  3. 2019Advances and Open Problems in Federated Learningcited by 3,021
  4. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  5. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  6. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,461
  7. 2020HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clientscited by 334
  8. 2021A Field Guide to Federated Optimizationcited by 312
  9. 2019Tighter Theory for Local SGD on Identical and Heterogeneous Datacited by 266
  10. 2019Model Pruning Enables Efficient Federated Learning on Edge Devicescited by 219
  11. 2020Federated Learning of a Mixture of Global and Local Modelscited by 208
  12. 2019SGD: General Analysis and Improved Ratescited by 206
38 more
  1. 2021FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropoutcited by 192
  2. 2018Expanding the Reach of Federated Learning by Reducing Client Resource Requirementscited by 162
  3. 2020Better Theory for SGD in the Nonconvex Worldcited by 156
  4. 2020On Biased Compression for Distributed Learningcited by 140
  5. 2022ProxSkip: Yes! Local Gradient Steps Provably Lead to Communication Acceleration! Finally!cited by 134
  6. 2021EF21: A New, Simpler, Theoretically Better, and Practically Faster Error Feedbackcited by 131
  7. 2020Dynamic Model Pruning with Feedbackcited by 125
  8. 2020Optimal Client Sampling for Federated Learningcited by 123
  9. 2019Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Modelcited by 116
  10. 2020PAGE: A Simple and Optimal Probabilistic Gradient Estimator for Nonconvex Optimizationcited by 116
  11. 2019Distributed Learning with Compressed Gradient Differencescited by 110
  12. 2022FedRolex: Model-Heterogeneous Federated Learning with Rolling Sub-Model Extractioncited by 102
  13. 2018SGD and Hogwild! Convergence Without the Bounded Gradients Assumptioncited by 94
  14. 2020Acceleration for Compressed Gradient Descent in Distributed and Federated Optimizationcited by 90
  15. 2019Natural Compression for Distributed Deep Learningcited by 88
  16. 2019A Unified Theory of SGD: Variance Reduction, Sampling, Quantization and Coordinate Descentcited by 84
  17. 2019Unified Optimal Analysis of the (Stochastic) Gradient Methodcited by 81
  18. 2019First Analysis of Local GD on Heterogeneous Datacited by 80
  19. 2020On the Convergence of SGD with Biased Gradientscited by 79
  20. 2020Variance-Reduced Methods for Machine Learningcited by 79
  21. 2021MARINA: Faster Non-Convex Distributed Learning with Compressioncited by 77
  22. 2020From Local SGD to Local Fixed-Point Methods for Federated Learningcited by 74
  23. 2019Don't Jump Through Hoops and Remove Those Loops: SVRG and Katyusha are Better Without the Outer Loopcited by 68
  24. 2019Scaling Distributed Machine Learning with In-Network Aggregationcited by 62
  25. 2023High-Probability Bounds for Stochastic Optimization and Variational Inequalities: the Case of Unbounded Variancecited by 60
  26. 2021Federated Dropout -- A Simple Approach for Enabling Federated Learning on Resource Constrained Devicescited by 56
  27. 2018Distributed learning with compressed gradientscited by 53
  28. 2021FedNL: Making Newton-Type Methods Applicable to Federated Learningcited by 50
  29. 2017Momentum and Stochastic Momentum for Stochastic Gradient, Newton, Proximal Point and Subspace Descent Methodscited by 49
  30. 2019Stochastic Sign Descent Methods: New Algorithms and Better Theorycited by 49
  31. 2018A Tight Convergence Analysis for Stochastic Gradient Descent with Delayed Updatescited by 47
  32. 2020Uncertainty Principle for Communication Compression in Distributed and Federated Learning and the Search for an Optimal Compressorcited by 45
  33. 2020Error Compensated Distributed SGD Can Be Acceleratedcited by 45
  34. 2022ZeroFL: Efficient On-Device Training for Federated Learning with Local Sparsitycited by 45
  35. 2020Optimal and Practical Algorithms for Smooth and Strongly Convex Decentralized Optimizationcited by 43
  36. 2020A Better Alternative to Error Feedback for Communication-Efficient Distributed Learningcited by 42
  37. 2020A Linearly Convergent Algorithm for Decentralized Optimization: Sending Less Bits for Free!cited by 41
  38. 2016Randomized Distributed Mean Estimation: Accuracy vs Communicationcited by 39

50 most cited of 187 All 187 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 4; 2025: 1.

6 papers.

  1. 2024FIARSE: Model-Heterogeneous Federated Learning via Importance-Aware Submodel Extractioncited by 21
  2. 2024FedP3: Federated Personalized and Privacy-friendly Network Pruning under Model Heterogeneitycited by 13
  3. 2024Sparser, Better, Deeper, Stronger: Improving Sparse Training with Exact Orthogonal Initializationcited by 3
  4. 2023Federated Learning over Hierarchical Wireless Networks: Training Latency Minimization via Submodel Partitioningcited by 1
  5. 2025On the Convergence and Stability of Distributed Sub-model Trainingcited by 1
  6. 2024MSfusion: A Dynamic Model Splitting Approach for Resource-Constrained Machines to Collaboratively Train Larger Modelscited by 0