# Reducing Communication for Split Learning by Randomized Top-k Sparsification

arXiv: https://arxiv.org/abs/2305.18469 · HTML: https://mlsanity.com/paper/2305.18469/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 55 references that are in the map.
- First posted on arXiv 29 May 2023.

## Builds on (most cited first, 28 of 28)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [Federated Learning: Challenges, Methods, and Future Directions](https://mlsanity.com/paper/1908.07873/) (2019)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [Deep Leakage from Gradients](https://mlsanity.com/paper/1906.08935/) (2019)
- [Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization](https://mlsanity.com/paper/2007.07481/) (2020)
- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [Session-based Recommendations with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06939/) (2015)
- [Exploring Generalization in Deep Learning](https://mlsanity.com/paper/1706.08947/) (2017)
- [Federated Learning with Matched Averaging](https://mlsanity.com/paper/2002.06440/) (2020)
- [Robust and Communication-Efficient Federated Learning from Non-IID Data](https://mlsanity.com/paper/1903.02891/) (2019)
- [Split learning for health: Distributed deep learning without sharing raw patient data](https://mlsanity.com/paper/1812.00564/) (2018)
- [Sparsified SGD with Memory](https://mlsanity.com/paper/1809.07599/) (2018)
- [Regularisation of Neural Networks by Enforcing Lipschitz Continuity](https://mlsanity.com/paper/1804.04368/) (2018)
- [Norm-Based Capacity Control in Neural Networks](https://mlsanity.com/paper/1503.00036/) (2015)
- [TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning](https://mlsanity.com/paper/1705.07878/) (2017)
- [Sparse Communication for Distributed Gradient Descent](https://mlsanity.com/paper/1704.05021/) (2017)
- [PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization](https://mlsanity.com/paper/1905.13727/) (2019)
- [ATOMO: Communication-efficient Learning via Atomic Sparsification](https://mlsanity.com/paper/1806.04090/) (2018)
- [Vertically Federated Graph Neural Network for Privacy-Preserving Node Classification](https://mlsanity.com/paper/2005.11903/) (2020)
- [Sparse Binary Compression: Towards Distributed Deep Learning with minimal Communication](https://mlsanity.com/paper/1805.08768/) (2018)
- [Compressed-VFL: Communication-Efficient Learning with Vertically Partitioned Data](https://mlsanity.com/paper/2206.08330/) (2022)
- [When Homomorphic Encryption Marries Secret Sharing: Secure Large-Scale Sparse Logistic Regression and Applications in Risk Control](https://mlsanity.com/paper/2008.08753/) (2020)
- [Accelerated Federated Learning with Decoupled Adaptive Optimization](https://mlsanity.com/paper/2207.07223/) (2022)
- [Communication and Computation Reduction for Split Learning using Asynchronous Training](https://mlsanity.com/paper/2107.09786/) (2021)
- [Communication Optimization in Large Scale Federated Learning using Autoencoder Compressed Weight Updates](https://mlsanity.com/paper/2108.05670/) (2021)

## Built on by (most cited first, 22 of 22)

- [Mobile Edge Intelligence for Large Language Models: A Contemporary Survey](https://mlsanity.com/paper/2407.18921/) (2024)
- [Communication-Efficient Split Learning via Adaptive Feature-Wise Compression](https://mlsanity.com/paper/2307.10805/) (2023)
- [Synergizing Foundation Models and Federated Learning: A Survey](https://mlsanity.com/paper/2406.12844/) (2024)
- [Quantized-Tinyllava: a new multimodal foundation model enables efficient split learning](https://mlsanity.com/paper/2511.23402/) (2025)
- [Towards Straggler-Resilient Split Federated Learning: An Unbalanced Update Approach](https://mlsanity.com/paper/2510.21155/) (2025)
- [NSC-SL: A Bandwidth-Aware Neural Subspace Compression for Communication-Efficient Split Learning](https://mlsanity.com/paper/2602.02696/) (2026)
- [Defending Label Inference Attacks in Split Learning under Regression Setting](https://mlsanity.com/paper/2308.09448/) (2023)
- [Mask-Encoded Sparsification: Mitigating Biased Gradients in Communication-Efficient Split Learning](https://mlsanity.com/paper/2408.13787/) (2024)
- [Ampere: Communication-Efficient and High-Accuracy Split Federated Learning](https://mlsanity.com/paper/2507.07130/) (2025)
- [Communication Efficient Split Learning of ViTs with Attention-based Double Compression](https://mlsanity.com/paper/2509.15058/) (2025)
- [Lean Clients, Full Accuracy: Hybrid Zeroth- and First-Order Split Federated Learning](https://mlsanity.com/paper/2601.09076/) (2026)
- [The Robustness of Spiking Neural Networks in Communication and its Application towards Network Efficiency in Federated Learning](https://mlsanity.com/paper/2409.12769/) (2024)
- [Oops!... They Stole it Again: Attacks on Split Learning](https://mlsanity.com/paper/2508.10598/) (2025)
- [SL-FAC: A Communication-Efficient Split Learning Framework with Frequency-Aware Compression](https://mlsanity.com/paper/2604.07316/) (2026)
- [AutoEncoder-Compressed Parallel Split Learning for Pre-trained Model Fine-Tuning](https://mlsanity.com/paper/2607.17913/) (2026)
- [Communication and Computation Efficient Split Federated Learning in O-RAN](https://mlsanity.com/paper/2508.02534/) (2025)
- [Exploiting Adaptive Channel Pruning for Communication-Efficient Split Learning](https://mlsanity.com/paper/2603.09792/) (2026)
- [LightSplit: Practical Privacy-Preserving Split Learning via Orthogonal Projections](https://mlsanity.com/paper/2605.13265/) (2026)
- [A Privacy Study of Sparse Collaborative Inference](https://mlsanity.com/paper/2608.16236/) (2026)
- [SplitLite: Low-Rank Residual Compression for Split Learning](https://mlsanity.com/paper/2608.23018/) (2026)
- [Contribution-Aware Bandwidth Allocation for Multimodal Split Learning](https://mlsanity.com/paper/2609.01406/) (2026)
- [Importance-Aware Feature Sparsification for Wireless Split Learning](https://mlsanity.com/paper/2609.39194/) (2026)

## Abstract (arXiv)

Split learning is a simple solution for Vertical Federated Learning (VFL), which has drawn substantial attention in both research and application due to its simplicity and efficiency. However, communication efficiency is still a crucial issue for split learning. In this paper, we investigate multiple communication reduction methods for split learning, including cut layer size reduction, top-k sparsification, quantization, and L1 regularization. Through analysis of the cut layer size reduction and top-k sparsification, we further propose randomized top-k sparsification, to make the model generalize and converge better. This is done by selecting top-k elements with a large probability while also having a small probability to select non-top-k elements. Empirical results show that compared with other communication-reduction methods, our proposed randomized top-k sparsification achieves a better model performance under the same compression level.