# Aggregating Capacity in FL through Successive Layer Training for Computationally-Constrained Devices

arXiv: https://arxiv.org/abs/2305.17005 · HTML: https://mlsanity.com/paper/2305.17005/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 44 references that are in the map.
- First posted on arXiv 26 May 2023.

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification](https://mlsanity.com/paper/1909.06335/) (2019)
- [LEAF: A Benchmark for Federated Settings](https://mlsanity.com/paper/1812.01097/) (2018)
- [HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clients](https://mlsanity.com/paper/2010.01264/) (2020)
- [Asynchronous Federated Optimization](https://mlsanity.com/paper/1903.03934/) (2019)
- [Applied Federated Learning: Improving Google Keyboard Query Suggestions](https://mlsanity.com/paper/1812.02903/) (2018)
- [FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout](https://mlsanity.com/paper/2102.13451/) (2021)
- [FedHealth: A Federated Transfer Learning Framework for Wearable Healthcare](https://mlsanity.com/paper/1907.09173/) (2019)
- [FetchSGD: Communication-Efficient Federated Learning with Sketching](https://mlsanity.com/paper/2007.07682/) (2020)
- [Expanding the Reach of Federated Learning by Reducing Client Resource Requirements](https://mlsanity.com/paper/1812.07210/) (2018)
- [FedRolex: Model-Heterogeneous Federated Learning with Rolling Sub-Model Extraction](https://mlsanity.com/paper/2212.01548/) (2022)
- [TiFL: A Tier-based Federated Learning System](https://mlsanity.com/paper/2001.09249/) (2020)
- [Putting An End to End-to-End: Gradient-Isolated Learning of Representations](https://mlsanity.com/paper/1905.11786/) (2019)
- [Lifelong Federated Reinforcement Learning: A Learning Architecture for Navigation in Cloud Robotic Systems](https://mlsanity.com/paper/1901.06455/) (2019)
- [LoCo: Local Contrastive Representation Learning](https://mlsanity.com/paper/2008.01342/) (2020)
- [Dynamic Tensor Rematerialization](https://mlsanity.com/paper/2006.09616/) (2020)
- [ZeroFL: Efficient On-Device Training for Federated Learning with Local Sparsity](https://mlsanity.com/paper/2208.02507/) (2022)
- [Accelerating Convolutional Neural Networks via Activation Map Compression](https://mlsanity.com/paper/1812.04056/) (2018)
- [FedHM: Efficient Federated Learning for Heterogeneous Models via Low-rank Factorization](https://mlsanity.com/paper/2111.14655/) (2021)
- [ProgFed: Effective, Communication, and Computation Efficient Federated Learning by Progressive Training](https://mlsanity.com/paper/2110.05323/) (2021)
- [A Federated Learning Framework for Healthcare IoT devices](https://mlsanity.com/paper/2005.05083/) (2020)
- [Towards Fair Federated Recommendation Learning: Characterizing the Inter-Dependence of System and Data Heterogeneity](https://mlsanity.com/paper/2206.02633/) (2022)
- [Forward Thinking: Building and Training Neural Networks One Layer at a Time](https://mlsanity.com/paper/1706.02480/) (2017)
- [DISTREAL: Distributed Resource-Aware Learning in Heterogeneous Systems](https://mlsanity.com/paper/2112.08761/) (2021)
- [Federated Learning for Localization: A Privacy-Preserving Crowdsourcing Method](https://mlsanity.com/paper/2001.01911/) (2020)
- [Federated Imitation Learning: A Privacy Considered Imitation Learning Framework for Cloud Robotic Systems with Heterogeneous Sensor Data](https://mlsanity.com/paper/1909.00895/) (2019)
- [FedNet2Net: Saving Communication and Computations in Federated Learning with Model Growing](https://mlsanity.com/paper/2207.09568/) (2022)

## Built on by (most cited first, 7 of 7)

- [Advances in Robust Federated Learning: A Survey with Heterogeneity Considerations](https://mlsanity.com/paper/2405.09839/) (2024)
- [FedSPU: Personalized Federated Learning for Resource-constrained Devices with Stochastic Parameter Update](https://mlsanity.com/paper/2403.11464/) (2024)
- [Efficient Federated Finetuning of Tiny Transformers with Resource-Constrained Devices](https://mlsanity.com/paper/2411.07826/) (2024)
- [Accelerated Training on Low-Power Edge Devices](https://mlsanity.com/paper/2502.18323/) (2025)
- [Bridging Generalization Gap of Heterogeneous Federated Clients Using Generative Models](https://mlsanity.com/paper/2508.01669/) (2025)
- [Energy and Memory-Efficient Federated Learning With Ordered Layer Freezing](https://mlsanity.com/paper/2512.23200/) (2025)
- [Efficient Zeroth-Order Federated Finetuning of Language Models on Resource-Constrained Devices](https://mlsanity.com/paper/2502.10239/) (2025)

## Abstract (arXiv)

Federated learning (FL) is usually performed on resource-constrained edge devices, e.g., with limited memory for the computation. If the required memory to train a model exceeds this limit, the device will be excluded from the training. This can lead to a lower accuracy as valuable data and computation resources are excluded from training, also causing bias and unfairness. The FL training process should be adjusted to such constraints. The state-of-the-art techniques propose training subsets of the FL model at constrained devices, reducing their resource requirements for training. But these techniques largely limit the co-adaptation among parameters of the model and are highly inefficient, as we show: it is actually better to train a smaller (less accurate) model by the system where all the devices can train the model end-to-end, than applying such techniques. We propose a new method that enables successive freezing and training of the parameters of the FL model at devices, reducing the training's resource requirements at the devices, while still allowing enough co-adaptation between parameters. We show through extensive experimental evaluation that our technique greatly improves the accuracy of the trained model (by 52.4 p.p.) compared with the state of the art, efficiently aggregating the computation capacity available on distributed devices.