# Server-Side Local Gradient Averaging and Learning Rate Acceleration for Scalable Split Learning

arXiv: https://arxiv.org/abs/2112.05929 · HTML: https://mlsanity.com/paper/2112.05929/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 25 references that are in the map.
- First posted on arXiv 11 Dec 2021.

## Builds on (most cited first, 23 of 23)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [Inverting Gradients -- How easy is it to break privacy in federated learning?](https://mlsanity.com/paper/2003.14053/) (2020)
- [Split learning for health: Distributed deep learning without sharing raw patient data](https://mlsanity.com/paper/1812.00564/) (2018)
- [Don't Decay the Learning Rate, Increase the Batch Size](https://mlsanity.com/paper/1711.00489/) (2017)
- [SplitFed: When Federated Learning Meets Split Learning](https://mlsanity.com/paper/2004.12088/) (2020)
- [Distributed learning of deep neural network over multiple agents](https://mlsanity.com/paper/1810.06060/) (2018)
- [One weird trick for parallelizing convolutional neural networks](https://mlsanity.com/paper/1404.5997/) (2014)
- [Wireless Network Intelligence at the Edge](https://mlsanity.com/paper/1812.02858/) (2018)
- [Larger-Scale Transformers for Multilingual Masked Language Modeling](https://mlsanity.com/paper/2105.00572/) (2021)
- [The History Began from AlexNet: A Comprehensive Survey on Deep Learning Approaches](https://mlsanity.com/paper/1803.01164/) (2018)
- [Revisiting Locally Supervised Learning: an Alternative to End-to-end Training](https://mlsanity.com/paper/2101.10832/) (2021)
- [Communication-Efficient and Distributed Learning Over Wireless Networks: Principles and Applications](https://mlsanity.com/paper/2008.02608/) (2020)
- [A Framework For Contrastive Self-Supervised Learning And Designing A New Approach](https://mlsanity.com/paper/2009.00104/) (2020)
- [Faithful Edge Federated Learning: Scalability and Privacy](https://mlsanity.com/paper/2106.15905/) (2021)

## Built on by (most cited first, 23 of 23)

- [Efficient Parallel Split Learning over Resource-constrained Wireless Edge Networks](https://mlsanity.com/paper/2303.15991/) (2023)
- [AdaptSFL: Adaptive Split Federated Learning in Resource-constrained Edge Networks](https://mlsanity.com/paper/2403.13101/) (2024)
- [HSplitLoRA: A Heterogeneous Split Parameter-Efficient Fine-Tuning Framework for Large Language Models](https://mlsanity.com/paper/2505.02795/) (2025)
- [Combined Federated and Split Learning in Edge Computing for Ubiquitous Intelligence in Internet of Things: State of the Art and Future Directions](https://mlsanity.com/paper/2207.09611/) (2022)
- [MergeSFL: Split Federated Learning with Feature Merging and Batch Size Regulation](https://mlsanity.com/paper/2311.13348/) (2023)
- [Visual Transformer Meets CutMix for Improved Accuracy, Communication Efficiency, and Data Privacy in Split Learning](https://mlsanity.com/paper/2207.00234/) (2022)
- [Federated Split Learning with Model Pruning and Gradient Quantization in Wireless Networks](https://mlsanity.com/paper/2412.06414/) (2024)
- [Local Learning on Transformers via Feature Reconstruction](https://mlsanity.com/paper/2212.14215/) (2022)
- [Breaking the Memory Wall for Heterogeneous Federated Learning via Model Splitting](https://mlsanity.com/paper/2410.11577/) (2024)
- [PFSL: Personalized & Fair Split Learning with Data & Label Privacy for thin clients](https://mlsanity.com/paper/2303.10624/) (2023)
- [SLPerf: a Unified Framework for Benchmarking Split Learning](https://mlsanity.com/paper/2304.01502/) (2023)
- [Parallel Split Learning with Global Sampling](https://mlsanity.com/paper/2407.15738/) (2024)
- [PiPar: Pipeline Parallelism for Collaborative Machine Learning](https://mlsanity.com/paper/2302.12803/) (2022)
- [Privacy-Preserving Split Learning with Vision Transformers using Patch-Wise Random and Noisy CutMix](https://mlsanity.com/paper/2408.01040/) (2024)
- [P3SL: Personalized Privacy-Preserving Split Learning on Heterogeneous Edge Devices](https://mlsanity.com/paper/2507.17228/) (2025)
- [Fine-tuning Multimodal Transformers on Edge: A Parallel Split Learning Approach](https://mlsanity.com/paper/2502.06355/) (2025)
- [CycleSL: Server-Client Cyclical Update Driven Scalable Split Learning](https://mlsanity.com/paper/2511.18611/) (2025)
- [Non-Federated Multi-Task Split Learning for Heterogeneous Sources](https://mlsanity.com/paper/2406.00150/) (2024)
- [Traversal Learning: A Lossless And Efficient Distributed Learning Framework](https://mlsanity.com/paper/2504.07471/) (2025)
- [Enhancing Split Learning with Sharded and Blockchain-Enabled SplitFed Approaches](https://mlsanity.com/paper/2509.25555/) (2025)
- [GAPSL: A Gradient-Aligned Parallel Split Learning over Data-Heterogeneous Edge Computing Systems](https://mlsanity.com/paper/2603.18540/) (2026)
- [A Survey on Split Learning for LLM Fine-Tuning: Models, Systems, and Privacy Optimizations](https://mlsanity.com/paper/2604.24468/) (2026)
- [Latency-Aware Client Assignment for Parallel Split Learning With Global Sampling](https://mlsanity.com/paper/2609.32132/) (2026)

## Abstract (arXiv)

In recent years, there have been great advances in the field of decentralized learning with private data. Federated learning (FL) and split learning (SL) are two spearheads possessing their pros and cons, and are suited for many user clients and large models, respectively. To enjoy both benefits, hybrid approaches such as SplitFed have emerged of late, yet their fundamentals have still been illusive. In this work, we first identify the fundamental bottlenecks of SL, and thereby propose a scalable SL framework, coined SGLR. The server under SGLR broadcasts a common gradient averaged at the split-layer, emulating FL without any additional communication across clients as opposed to SplitFed. Meanwhile, SGLR splits the learning rate into its server-side and client-side rates, and separately adjusts them to support many clients in parallel. Simulation results corroborate that SGLR achieves higher accuracy than other baseline SL methods including SplitFed, which is even on par with FL consuming higher energy and communication costs. As a secondary result, we observe greater reduction in leakage of sensitive information via mutual information using SLGR over the baselines.