# ProgFed: Effective, Communication, and Computation Efficient Federated Learning by Progressive Training

arXiv: https://arxiv.org/abs/2110.05323 · HTML: https://mlsanity.com/paper/2110.05323/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 66 references that are in the map.
- First posted on arXiv 11 Oct 2021.

## Builds on (most cited first, 30 of 44)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [Improved Regularization of Convolutional Neural Networks with Cutout](https://mlsanity.com/paper/1708.04552/) (2017)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [Federated Learning: Strategies for Improving Communication Efficiency](https://mlsanity.com/paper/1610.05492/) (2016)
- [Adaptive Federated Optimization](https://mlsanity.com/paper/2003.00295/) (2020)
- [Ensemble Distillation for Robust Model Fusion in Federated Learning](https://mlsanity.com/paper/2006.07242/) (2020)
- [Learning Efficient Convolutional Networks through Network Slimming](https://mlsanity.com/paper/1708.06519/) (2017)
- [BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks](https://mlsanity.com/paper/1709.01686/) (2017)
- [signSGD: Compressed Optimisation for Non-Convex Problems](https://mlsanity.com/paper/1802.04434/) (2018)
- [Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challenge](https://mlsanity.com/paper/1811.02629/) (2018)
- [Patient Knowledge Distillation for BERT Model Compression](https://mlsanity.com/paper/1908.09355/) (2019)
- [Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training](https://mlsanity.com/paper/1712.01887/) (2017)
- [FedMD: Heterogenous Federated Learning via Model Distillation](https://mlsanity.com/paper/1910.03581/) (2019)
- [Sparsified SGD with Memory](https://mlsanity.com/paper/1809.07599/) (2018)
- [Error Feedback Fixes SignSGD and other Gradient Compression Schemes](https://mlsanity.com/paper/1901.09847/) (2019)
- [Slimmable Neural Networks](https://mlsanity.com/paper/1812.08928/) (2018)
- [The Non-IID Data Quagmire of Decentralized Machine Learning](https://mlsanity.com/paper/1910.00189/) (2019)
- [Model compression via distillation and quantization](https://mlsanity.com/paper/1802.05668/) (2018)
- [TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning](https://mlsanity.com/paper/1705.07878/) (2017)
- [Dynamic Model Pruning with Feedback](https://mlsanity.com/paper/2006.07253/) (2020)
- [DoubleSqueeze: Parallel Stochastic Gradient Descent with Double-Pass Error-Compensated Compression](https://mlsanity.com/paper/1905.05957/) (2019)

## Built on by (most cited first, 27 of 27)

- [Heterogeneous Federated Learning: State-of-the-art and Research Challenges](https://mlsanity.com/paper/2307.10616/) (2023)
- [FedTune: A Deep Dive into Efficient Federated Fine-Tuning with Pre-trained Transformers](https://mlsanity.com/paper/2211.08025/) (2022)
- [Accelerated Federated Learning with Decoupled Adaptive Optimization](https://mlsanity.com/paper/2207.07223/) (2022)
- [Federated Learning for Computer Vision](https://mlsanity.com/paper/2308.13558/) (2023)
- [BAFFLE: A Baseline of Backpropagation-Free Federated Learning](https://mlsanity.com/paper/2301.12195/) (2023)
- [ProGAP: Progressive Graph Neural Networks with Differential Privacy Guarantees](https://mlsanity.com/paper/2304.08928/) (2023)
- [Breaking the Memory Wall for Heterogeneous Federated Learning via Progressive Training](https://mlsanity.com/paper/2404.13349/) (2024)
- [Aggregating Capacity in FL through Successive Layer Training for Computationally-Constrained Devices](https://mlsanity.com/paper/2305.17005/) (2023)
- [Federated Learning for Inference at Anytime and Anywhere](https://mlsanity.com/paper/2212.04084/) (2022)
- [Why Go Full? Elevating Federated Learning Through Partial Network Updates](https://mlsanity.com/paper/2410.11559/) (2024)
- [Aequa: Fair Model Rewards in Collaborative Learning via Slimmable Networks](https://mlsanity.com/paper/2502.04850/) (2025)
- [FedLAP-DP: Federated Learning by Sharing Differentially Private Loss Approximations](https://mlsanity.com/paper/2302.01068/) (2023)
- [Language Models as Zero-shot Lossless Gradient Compressors: Towards General Neural Parameter Prior Models](https://mlsanity.com/paper/2409.17836/) (2024)
- [A Thorough Assessment of the Non-IID Data Impact in Federated Learning](https://mlsanity.com/paper/2503.17070/) (2025)
- [Learning Like Humans: Resource-Efficient Federated Fine-Tuning through Cognitive Developmental Stages](https://mlsanity.com/paper/2508.00041/) (2025)
- [FLEdge: Benchmarking Federated Machine Learning Applications in Edge Computing Systems](https://mlsanity.com/paper/2306.05172/) (2023)
- [FedCode: Communication-Efficient Federated Learning via Transferring Codebooks](https://mlsanity.com/paper/2311.09270/) (2023)
- [MoSE: Mixture of Slimmable Experts for Efficient and Adaptive Language Models](https://mlsanity.com/paper/2602.06154/) (2026)
- [Understanding Progressive Training Through the Framework of Randomized Coordinate Descent](https://mlsanity.com/paper/2306.03626/) (2023)
- [FedNet2Net: Saving Communication and Computations in Federated Learning with Model Growing](https://mlsanity.com/paper/2207.09568/) (2022)
- [VeryFL: A Verify Federated Learning Framework Embedded with Blockchain](https://mlsanity.com/paper/2311.15617/) (2023)
- [Resource-Efficient Federated Multimodal Learning via Layer-wise and Progressive Training](https://mlsanity.com/paper/2407.15426/) (2024)
- [Efficient Federated Finetuning of Tiny Transformers with Resource-Constrained Devices](https://mlsanity.com/paper/2411.07826/) (2024)
- [FedCliP: Federated Learning with Client Pruning](https://mlsanity.com/paper/2301.06768/) (2023)
- [Federated Hyperdimensional Computing](https://mlsanity.com/paper/2312.15966/) (2023)
- [Resource-efficient Layer-wise Federated Self-supervised Learning](https://mlsanity.com/paper/2401.11647/) (2024)
- [Efficient Zeroth-Order Federated Finetuning of Language Models on Resource-Constrained Devices](https://mlsanity.com/paper/2502.10239/) (2025)

## Abstract (arXiv)

Federated learning is a powerful distributed learning scheme that allows numerous edge devices to collaboratively train a model without sharing their data. However, training is resource-intensive for edge devices, and limited network bandwidth is often the main bottleneck. Prior work often overcomes the constraints by condensing the models or messages into compact formats, e.g., by gradient compression or distillation. In contrast, we propose ProgFed, the first progressive training framework for efficient and effective federated learning. It inherently reduces computation and two-way communication costs while maintaining the strong performance of the final models. We theoretically prove that ProgFed converges at the same asymptotic rate as standard training on full models. Extensive results on a broad range of architectures, including CNNs (VGG, ResNet, ConvNets) and U-nets, and diverse tasks from simple classification to medical image segmentation show that our highly effective training approach saves up to 20% computation and up to 63% communication costs for converged models. As our approach is also complimentary to prior work on compression, we can achieve a wide range of trade-offs by combining these techniques, showing reduced communication of up to 50× at only 0.1% loss in utility. Code is available at https://github.com/hui-po-wang/ProgFed.