# ZeroFL: Efficient On-Device Training for Federated Learning with Local Sparsity

arXiv: https://arxiv.org/abs/2208.02507 · HTML: https://mlsanity.com/paper/2208.02507/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 63 references that are in the map.
- First posted on arXiv 4 Aug 2022.

## Builds on (most cited first, 30 of 44)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [Federated Learning: Strategies for Improving Communication Efficiency](https://mlsanity.com/paper/1610.05492/) (2016)
- [Adaptive Federated Optimization](https://mlsanity.com/paper/2003.00295/) (2020)
- [Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification](https://mlsanity.com/paper/1909.06335/) (2019)
- [LEAF: A Benchmark for Federated Settings](https://mlsanity.com/paper/1812.01097/) (2018)
- [Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition](https://mlsanity.com/paper/1804.03209/) (2018)
- [Channel Pruning for Accelerating Very Deep Neural Networks](https://mlsanity.com/paper/1707.06168/) (2017)
- [Ensemble Distillation for Robust Model Fusion in Federated Learning](https://mlsanity.com/paper/2006.07242/) (2020)
- [Federated Learning for Mobile Keyboard Prediction](https://mlsanity.com/paper/1811.03604/) (2018)
- [Importance Estimation for Neural Network Pruning](https://mlsanity.com/paper/1906.10771/) (2019)
- [Federated Learning with Personalization Layers](https://mlsanity.com/paper/1912.00818/) (2019)
- [Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks](https://mlsanity.com/paper/2102.00554/) (2021)
- [Flower: A Friendly Federated Learning Research Framework](https://mlsanity.com/paper/2007.14390/) (2020)
- [ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression](https://mlsanity.com/paper/1707.06342/) (2017)
- [Bayesian Nonparametric Federated Learning of Neural Networks](https://mlsanity.com/paper/1905.12022/) (2019)
- [Data-Free Knowledge Distillation for Heterogeneous Federated Learning](https://mlsanity.com/paper/2105.10056/) (2021)
- [Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks](https://mlsanity.com/paper/1808.06866/) (2018)
- [Variational Dropout Sparsifies Deep Neural Networks](https://mlsanity.com/paper/1701.05369/) (2017)
- [Dynamic Network Surgery for Efficient DNNs](https://mlsanity.com/paper/1608.04493/) (2016)
- [NISP: Pruning Networks using Neuron Importance Score Propagation](https://mlsanity.com/paper/1711.05908/) (2017)
- [Model Pruning Enables Efficient Federated Learning on Edge Devices](https://mlsanity.com/paper/1909.12326/) (2019)
- [FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout](https://mlsanity.com/paper/2102.13451/) (2021)
- [Expanding the Reach of Federated Learning by Reducing Client Resource Requirements](https://mlsanity.com/paper/1812.07210/) (2018)
- [ATOMO: Communication-efficient Learning via Atomic Sparsification](https://mlsanity.com/paper/1806.04090/) (2018)
- [Hello Edge: Keyword Spotting on Microcontrollers](https://mlsanity.com/paper/1711.07128/) (2017)

## Built on by (most cited first, 30 of 45)

- [SLoRA: Federated Parameter Efficient Fine-Tuning of Language Models](https://mlsanity.com/paper/2308.06522/) (2023)
- [Federated Learning for Computationally-Constrained Heterogeneous Devices: A Survey](https://mlsanity.com/paper/2307.09182/) (2023)
- [Efficient Personalized Federated Learning via Sparse Model-Adaptation](https://mlsanity.com/paper/2305.02776/) (2023)
- [Federated LoRA with Sparse Communication](https://mlsanity.com/paper/2406.05233/) (2024)
- [On the Convergence of Zeroth-Order Federated Tuning for Large Language Models](https://mlsanity.com/paper/2402.05926/) (2024)
- [Multisize Dataset Condensation](https://mlsanity.com/paper/2403.06075/) (2024)
- [FedImpro: Measuring and Improving Client Update in Federated Learning](https://mlsanity.com/paper/2402.07011/) (2024)
- [FedMef: Towards Memory-efficient Federated Dynamic Pruning](https://mlsanity.com/paper/2403.14737/) (2024)
- [Non-IID data in Federated Learning: A Survey with Taxonomy, Metrics, Methods, Frameworks and Future Directions](https://mlsanity.com/paper/2411.12377/) (2024)
- [Lottery Aware Sparsity Hunting: Enabling Federated Learning on Resource-Limited Edge](https://mlsanity.com/paper/2208.13092/) (2022)
- [Distributed Pruning Towards Tiny Neural Networks in Federated Learning](https://mlsanity.com/paper/2212.01977/) (2022)
- [FuseFL: One-Shot Federated Learning through the Lens of Causality with Progressive Model Fusion](https://mlsanity.com/paper/2410.20380/) (2024)
- [SpaFL: Communication-Efficient Federated Learning with Sparse Models and Low computational Overhead](https://mlsanity.com/paper/2406.00431/) (2024)
- [FedBAT: Communication-Efficient Federated Learning via Learnable Binarization](https://mlsanity.com/paper/2408.03215/) (2024)
- [Aggregating Capacity in FL through Successive Layer Training for Computationally-Constrained Devices](https://mlsanity.com/paper/2305.17005/) (2023)
- [Federated Dynamical Low-Rank Training with Global Loss Convergence Guarantees](https://mlsanity.com/paper/2406.17887/) (2024)
- [Towards a Better Theoretical Understanding of Independent Subnetwork Training](https://mlsanity.com/paper/2306.16484/) (2023)
- [Sheaf HyperNetworks for Personalized Federated Learning](https://mlsanity.com/paper/2405.20882/) (2024)
- [The Panaceas for Improving Low-Rank Decomposition in Communication-Efficient Federated Learning](https://mlsanity.com/paper/2505.23176/) (2025)
- [Breaking Physical and Linguistic Borders: Multilingual Federated Prompt Tuning for Low-Resource Languages](https://mlsanity.com/paper/2507.03003/) (2025)
- [Secure Vertical Federated Learning Under Unreliable Connectivity](https://mlsanity.com/paper/2305.16794/) (2023)
- [Don't Memorize; Mimic The Past: Federated Class Incremental Learning Without Episodic Memory](https://mlsanity.com/paper/2307.00497/) (2023)
- [Masked Random Noise for Communication Efficient Federated Learning](https://mlsanity.com/paper/2408.03220/) (2024)
- [Bandwidth-Aware and Overlap-Weighted Compression for Communication-Efficient Federated Learning](https://mlsanity.com/paper/2408.14736/) (2024)
- [FedRTS: Federated Robust Pruning via Combinatorial Thompson Sampling](https://mlsanity.com/paper/2501.19122/) (2025)
- [Efficient Vertical Federated Learning with Secure Aggregation](https://mlsanity.com/paper/2305.11236/) (2023)
- [Evaluating Privacy Leakage in Split Learning](https://mlsanity.com/paper/2305.12997/) (2023)
- [FLoCoRA: Federated learning compression with low-rank adaptation](https://mlsanity.com/paper/2406.14082/) (2024)
- [Energy-Efficient Channel Decoding for Wireless Federated Learning: Convergence Analysis and Adaptive Design](https://mlsanity.com/paper/2407.13703/) (2024)
- [DAF: An Efficient End-to-End Dynamic Activation Framework for on-Device DNN Training](https://mlsanity.com/paper/2507.07149/) (2025)

## Abstract (arXiv)

When the available hardware cannot meet the memory and compute requirements to efficiently train high performing machine learning models, a compromise in either the training quality or the model complexity is needed. In Federated Learning (FL), nodes are orders of magnitude more constrained than traditional server-grade hardware and are often battery powered, severely limiting the sophistication of models that can be trained under this paradigm. While most research has focused on designing better aggregation strategies to improve convergence rates and in alleviating the communication costs of FL, fewer efforts have been devoted to accelerating on-device training. Such stage, which repeats hundreds of times (i.e. every round) and can involve thousands of devices, accounts for the majority of the time required to train federated models and, the totality of the energy consumption at the client side. In this work, we present the first study on the unique aspects that arise when introducing sparsity at training time in FL workloads. We then propose ZeroFL, a framework that relies on highly sparse operations to accelerate on-device training. Models trained with ZeroFL and 95% sparsity achieve up to 2.3% higher accuracy compared to competitive baselines obtained from adapting a state-of-the-art sparse training framework to the FL setting.