# Parallel Split Learning with Global Sampling

arXiv: https://arxiv.org/abs/2407.15738 · HTML: https://mlsanity.com/paper/2407.15738/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 50 references that are in the map.
- First posted on arXiv 22 Jul 2024.

## Builds on (most cited first, 24 of 24)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Split learning for health: Distributed deep learning without sharing raw patient data](https://mlsanity.com/paper/1812.00564/) (2018)
- [Client Selection for Federated Learning with Heterogeneous Resources in Mobile Edge](https://mlsanity.com/paper/1804.08333/) (2018)
- [SplitFed: When Federated Learning Meets Split Learning](https://mlsanity.com/paper/2004.12088/) (2020)
- [Edge Intelligence: Paving the Last Mile of Artificial Intelligence with Edge Computing](https://mlsanity.com/paper/1905.10083/) (2019)
- [Efficient Parallel Split Learning over Resource-constrained Wireless Edge Networks](https://mlsanity.com/paper/2303.15991/) (2023)
- [Wireless Network Intelligence at the Edge](https://mlsanity.com/paper/1812.02858/) (2018)
- [Speeding Up Distributed Machine Learning Using Codes](https://mlsanity.com/paper/1512.02673/) (2015)
- [Split Learning over Wireless Networks: Parallel Design and Resource Management](https://mlsanity.com/paper/2204.08119/) (2022)
- [Split Learning for collaborative deep learning in healthcare](https://mlsanity.com/paper/1912.12115/) (2019)
- [End-to-End Evaluation of Federated Learning and Split Learning for Internet of Things](https://mlsanity.com/paper/2003.13376/) (2020)
- [Advancements of federated learning towards privacy preservation: from federated learning to split learning](https://mlsanity.com/paper/2011.14818/) (2020)
- [On the Computational Inefficiency of Large Batch Sizes for Stochastic Gradient Descent](https://mlsanity.com/paper/1811.12941/) (2018)
- [Server-Side Local Gradient Averaging and Learning Rate Acceleration for Scalable Split Learning](https://mlsanity.com/paper/2112.05929/) (2021)
- [Convergence Analysis of Sequential Federated Learning on Heterogeneous Data](https://mlsanity.com/paper/2311.03154/) (2023)
- [A Bargaining Game for Personalized, Energy Efficient Split Learning over Wireless Networks](https://mlsanity.com/paper/2212.06107/) (2022)
- [Evaluation and Optimization of Distributed Machine Learning Techniques for Internet of Things](https://mlsanity.com/paper/2103.02762/) (2021)
- [Comparison of Privacy-Preserving Distributed Deep Learning Methods in Healthcare](https://mlsanity.com/paper/2012.12591/) (2020)
- [Distilling On-Device Intelligence at the Network Edge](https://mlsanity.com/paper/1908.05895/) (2019)
- [Resource-efficient Parallel Split Learning in Heterogeneous Edge Computing](https://mlsanity.com/paper/2403.15815/) (2024)

## Built on by (most cited first, 3 of 3)

- [GAPSL: A Gradient-Aligned Parallel Split Learning over Data-Heterogeneous Edge Computing Systems](https://mlsanity.com/paper/2603.18540/) (2026)
- [Concurrent Split Learning Through Stable Client Clustering](https://mlsanity.com/paper/2609.29395/) (2026)
- [Latency-Aware Client Assignment for Parallel Split Learning With Global Sampling](https://mlsanity.com/paper/2609.32132/) (2026)

## Abstract (arXiv)

Parallel split learning (PSL) suffers from two intertwined issues: the effective batch size grows with the number of clients, and data that is not identically and independently distributed (non-IID) skews global batches. We present parallel split learning with global sampling (GPSL), a server-driven scheme that fixes the global batch size while computing per-client batch-size schedules using pooled-level proportions. The actual samples are drawn locally without replacement by each selected client. This eliminates per-class rounding, decouples the effective batch from the client count, and makes each global batch distributionally equivalent to centralized uniform sampling without replacement. Consequently, we obtain finite-population deviation guarantees via Serfling's inequality, yielding a zero rounding bias compared to local sampling schemes. GPSL is a drop-in replacement for PSL with negligible overhead and scales to large client populations. In extensive experiments on CIFAR-10/100 and ResNet-18/34 under non-IID splits, GPSL stabilizes optimization and achieves centralized-like accuracy, while fixed local batching trails by up to 60%. Furthermore, GPSL shortens training time by avoiding inflation of training steps induced by data-depletion. These findings suggest GPSL is a promising and scalable approach for learning in resource-constrained environments.