mlsanityData to 14 Oct 2026

On the Convergence and Stability of Distributed Sub-model Training

arXiv 2511.06132 ↗First posted 8 Nov 2025cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

As learning models continue to grow in size, enabling on-device local training of these models has emerged as a critical challenge in federated learning. A popular solution is sub-model training, where the server only distributes randomly sampled sub-models to the edge clients, and clients only update these small models. However, those random sampling of sub-models may not give satisfying convergence performance. In this paper, observing the success of SGD with shuffling, we propose a distributed shuffled sub-model training, where the full model is partitioned into several sub-models in advance, and the server shuffles those sub-models, sends each of them to clients at each round, and by the end of local updating period, clients send back the updated sub-models, and server averages them. We establish the convergence rate of this algorithm. We also study the generalization of distributed sub-model training via stability analysis, and find that the sub-model training can improve the generalization via amplifying the stability of training process. The extensive experiments also validate our theoretical findings.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 32 of its 36 references that are in the map. First posted on arXiv 8 Nov 2025.

Builds on

32 of 36 references.

References · 32

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  2. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,229
  3. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 682
  4. 2018Local SGD Converges Fast and Communicates Littlecited by 605
  5. 2019FedMD: Heterogenous Federated Learning via Model Distillationcited by 459
  6. 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learningcited by 383
  7. 2020HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clientscited by 337
  8. 2019Tighter Theory for Local SGD on Identical and Heterogeneous Datacited by 266
  9. 2019Model Pruning Enables Efficient Federated Learning on Edge Devicescited by 222
  10. 2020FedBE: Making Bayesian Model Ensemble Applicable to Federated Learningcited by 176
  11. 2019On the Convergence of Local Descent Methods in Federated Learningcited by 153
  12. 2020Is Local SGD Better than Minibatch SGD?cited by 151
20 more
  1. 2020Minibatch vs Local SGD for Heterogeneous Distributed Learningcited by 147
  2. 2013Mini-batch Stochastic Approximation Methods for Nonconvex Stochastic Composite Optimizationcited by 143
  3. 2021FedPara: Low-Rank Hadamard Product for Communication-Efficient Federated Learningcited by 141
  4. 2022On the Effectiveness of Parameter-Efficient Fine-Tuningcited by 141
  5. 2019One-Shot Federated Learningcited by 125
  6. 2019Local SGD with Periodic Averaging: Tighter Analysis and Adaptive Synchronizationcited by 107
  7. 2022FedRolex: Model-Heterogeneous Federated Learning with Rolling Sub-Model Extractioncited by 102
  8. 2020Federated Accelerated Stochastic Gradient Descentcited by 85
  9. 2021FedAUX: Leveraging Unlabeled Auxiliary Data in Federated Learningcited by 69
  10. 2020Federated Model Distillation with Noise-Free Differential Privacycited by 40
  11. 2021FedHM: Efficient Federated Learning for Heterogeneous Models via Low-rank Factorizationcited by 31
  12. 2023On the Convergence of Federated Averaging with Cyclic Client Participationcited by 31
  13. 2023Understanding Generalization of Federated Learning via Stability: Heterogeneity Matterscited by 27
  14. 2023Every Parameter Matters: Ensuring the Convergence of Federated Learning with Dynamic Heterogeneous Models Reductioncited by 24
  15. 2024FIARSE: Model-Heterogeneous Federated Learning via Importance-Aware Submodel Extractioncited by 21
  16. 2021Masked Training of Neural Networks with Partial Gradientscited by 16
  17. 2021Communication-Efficient Federated Learning with Dual-Side Low-Rank Compressioncited by 10
  18. 2023Towards a Better Theoretical Understanding of Independent Subnetwork Trainingcited by 6
  19. 2023Distributed Personalized Empirical Risk Minimizationcited by 6
  20. 2023MAST: Model-Agnostic Sparsified Trainingcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Capacity Confounds and Coverage Guarantees in Adaptive Sub-model Federated Learningcited by 0