# FedLite: A Scalable Approach for Federated Learning on Resource-constrained Clients

arXiv: https://arxiv.org/abs/2201.11865 · HTML: https://mlsanity.com/paper/2201.11865/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 44 references that are in the map.
- First posted on arXiv 28 Jan 2022.

## Builds on (most cited first, 24 of 24)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Federated Learning: Challenges, Methods, and Future Directions](https://mlsanity.com/paper/1908.07873/) (2019)
- [Adaptive Federated Optimization](https://mlsanity.com/paper/2003.00295/) (2020)
- [signSGD: Compressed Optimisation for Non-Convex Problems](https://mlsanity.com/paper/1802.04434/) (2018)
- [Federated Learning with Differential Privacy: Algorithms and Performance Analysis](https://mlsanity.com/paper/1911.00222/) (2019)
- [Split learning for health: Distributed deep learning without sharing raw patient data](https://mlsanity.com/paper/1812.00564/) (2018)
- [FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantization](https://mlsanity.com/paper/1909.13014/) (2019)
- [HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clients](https://mlsanity.com/paper/2010.01264/) (2020)
- [A Field Guide to Federated Optimization](https://mlsanity.com/paper/2107.06917/) (2021)
- [SplitFed: When Federated Learning Meets Split Learning](https://mlsanity.com/paper/2004.12088/) (2020)
- [One weird trick for parallelizing convolutional neural networks](https://mlsanity.com/paper/1404.5997/) (2014)
- [PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization](https://mlsanity.com/paper/1905.13727/) (2019)
- [FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout](https://mlsanity.com/paper/2102.13451/) (2021)
- [QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding](https://mlsanity.com/paper/1610.02132/) (2016)
- [Minibatch vs Local SGD for Heterogeneous Distributed Learning](https://mlsanity.com/paper/2006.04735/) (2020)
- [Distributed Mean Estimation with Limited Communication](https://mlsanity.com/paper/1611.00429/) (2016)
- [ATOMO: Communication-efficient Learning via Atomic Sparsification](https://mlsanity.com/paper/1806.04090/) (2018)
- [InstaHide: Instance-hiding Schemes for Private Distributed Learning](https://mlsanity.com/paper/2010.02772/) (2020)
- [Differentiable Product Quantization for End-to-End Embedding Compression](https://mlsanity.com/paper/1908.09756/) (2019)
- [PyVertical: A Vertical Federated Learning Framework for Multi-headed SplitNN](https://mlsanity.com/paper/2104.00489/) (2021)
- [FedJAX: Federated learning simulation with JAX](https://mlsanity.com/paper/2108.02117/) (2021)
- [On the Utility of Gradient Compression in Distributed Training Systems](https://mlsanity.com/paper/2103.00543/) (2021)

## Built on by (most cited first, 15 of 15)

- [Communication-Efficient Split Learning via Adaptive Feature-Wise Compression](https://mlsanity.com/paper/2307.10805/) (2023)
- [FLuID: Mitigating Stragglers in Federated Learning using Invariant Dropout](https://mlsanity.com/paper/2307.02623/) (2023)
- [Combined Federated and Split Learning in Edge Computing for Ubiquitous Intelligence in Internet of Things: State of the Art and Future Directions](https://mlsanity.com/paper/2207.09611/) (2022)
- [Federated Learning While Providing Model as a Service: Joint Training and Inference Optimization](https://mlsanity.com/paper/2312.12863/) (2023)
- [Towards Straggler-Resilient Split Federated Learning: An Unbalanced Update Approach](https://mlsanity.com/paper/2510.21155/) (2025)
- [Reconciling Security and Communication Efficiency in Federated Learning](https://mlsanity.com/paper/2207.12779/) (2022)
- [SplitFedZip: Learned Compression for Data Transfer Reduction in Split-Federated Learning](https://mlsanity.com/paper/2412.17150/) (2024)
- [SCALA: Split Federated Learning with Concatenated Activations and Logit Adjustments](https://mlsanity.com/paper/2405.04875/) (2024)
- [GAS: Generative Activation-Aided Asynchronous Split Federated Learning](https://mlsanity.com/paper/2409.01251/) (2024)
- [Collaborative Distributed Machine Learning](https://mlsanity.com/paper/2309.16584/) (2023)
- [A Novel Federated Learning-Based IDS for Enhancing UAVs Privacy and Security](https://mlsanity.com/paper/2312.04135/) (2023)
- [Reducing Impacts of System Heterogeneity in Federated Learning using Weight Update Magnitudes](https://mlsanity.com/paper/2208.14808/) (2022)
- [A Survey on Split Learning for LLM Fine-Tuning: Models, Systems, and Privacy Optimizations](https://mlsanity.com/paper/2604.24468/) (2026)
- [SplitLite: Low-Rank Residual Compression for Split Learning](https://mlsanity.com/paper/2608.23018/) (2026)
- [Importance-Aware Feature Sparsification for Wireless Split Learning](https://mlsanity.com/paper/2609.39194/) (2026)

## Abstract (arXiv)

In classical federated learning, the clients contribute to the overall training by communicating local updates for the underlying model on their private data to a coordinating server. However, updating and communicating the entire model becomes prohibitively expensive when resource-constrained clients collectively aim to train a large machine learning model. Split learning provides a natural solution in such a setting, where only a small part of the model is stored and trained on clients while the remaining large part of the model only stays at the servers. However, the model partitioning employed in split learning introduces a significant amount of communication cost. This paper addresses this issue by compressing the additional communication using a novel clustering scheme accompanied by a gradient correction method. Extensive empirical evaluations on image and text benchmarks show that the proposed method can achieve up to 490× communication cost reduction with minimal drop in accuracy, and enables a desirable performance vs. communication trade-off.