Parallel Split Learning with Global Sampling
Not on the map. Why?
Abstract (arXiv)
Parallel split learning (PSL) suffers from two intertwined issues: the effective batch size grows with the number of clients, and data that is not identically and independently distributed (non-IID) skews global batches. We present parallel split learning with global sampling (GPSL), a server-driven scheme that fixes the global batch size while computing per-client batch-size schedules using pooled-level proportions. The actual samples are drawn locally without replacement by each selected client. This eliminates per-class rounding, decouples the effective batch from the client count, and makes each global batch distributionally equivalent to centralized uniform sampling without replacement. Consequently, we obtain finite-population deviation guarantees via Serfling's inequality, yielding a zero rounding bias compared to local sampling schemes. GPSL is a drop-in replacement for PSL with negligible overhead and scales to large client populations. In extensive experiments on CIFAR-10/100 and ResNet-18/34 under non-IID splits, GPSL stabilizes optimization and achieves centralized-like accuracy, while fixed local batching trails by up to 60%. Furthermore, GPSL shortens training time by avoiding inflation of training steps induced by data-depletion. These findings suggest GPSL is a promising and scalable approach for learning in resource-constrained environments.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 24 of its 50 references that are in the map. First posted on arXiv 22 Jul 2024.
Builds on
24 of 50 references.
References · 24
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
- 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
- 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
- 2018Group Normalizationcited by 1,514
- 2018Split learning for health: Distributed deep learning without sharing raw patient datacited by 391
- 2018Client Selection for Federated Learning with Heterogeneous Resources in Mobile Edgecited by 347
- 2020SplitFed: When Federated Learning Meets Split Learningcited by 308
- 2019Edge Intelligence: Paving the Last Mile of Artificial Intelligence with Edge Computingcited by 225
- 2023Efficient Parallel Split Learning over Resource-constrained Wireless Edge Networkscited by 95
- 2018Wireless Network Intelligence at the Edgecited by 90
- 2015Speeding Up Distributed Machine Learning Using Codescited by 76
12 more
- 2022Split Learning over Wireless Networks: Parallel Design and Resource Managementcited by 74
- 2019Split Learning for collaborative deep learning in healthcarecited by 72
- 2020End-to-End Evaluation of Federated Learning and Split Learning for Internet of Thingscited by 64
- 2020Advancements of federated learning towards privacy preservation: from federated learning to split learningcited by 34
- 2018On the Computational Inefficiency of Large Batch Sizes for Stochastic Gradient Descentcited by 25
- 2023Convergence Analysis of Sequential Federated Learning on Heterogeneous Datacited by 23
- 2021Server-Side Local Gradient Averaging and Learning Rate Acceleration for Scalable Split Learningcited by 22
- 2022A Bargaining Game for Personalized, Energy Efficient Split Learning over Wireless Networkscited by 22
- 2021Evaluation and Optimization of Distributed Machine Learning Techniques for Internet of Thingscited by 20
- 2020Comparison of Privacy-Preserving Distributed Deep Learning Methods in Healthcarecited by 14
- 2019Distilling On-Device Intelligence at the Network Edgecited by 5
- 2024Resource-efficient Parallel Split Learning in Heterogeneous Edge Computingcited by 3
Built on by
Citing papers per year: 2026: 3.
3 papers.