mlsanityData to 14 Oct 2026

Speed Up Federated Learning in Heterogeneous Environment: A Dynamic Tiering Approach

arXiv 2312.05642 ↗First posted 9 Dec 2023cs.LG cs.AI cs.MAcited by 3

Not on the map. Why?

Abstract (arXiv)

Federated learning (FL) enables collaboratively training a model while keeping the training data decentralized and private. However, one significant impediment to training a model using FL, especially large models, is the resource constraints of devices with heterogeneous computation and communication capacities as well as varying task sizes. Such heterogeneity would render significant variations in the training time of clients, resulting in a longer overall training time as well as a waste of resources in faster clients. To tackle these heterogeneity issues, we propose the Dynamic Tiering-based Federated Learning (DTFL) system where slower clients dynamically offload part of the model to the server to alleviate resource constraints and speed up training. By leveraging the concept of Split Learning, DTFL offloads different portions of the global model to clients in different tiers and enables each client to update the models in parallel via local-loss-based training. This helps reduce the computation and communication demand on resource-constrained devices and thus mitigates the straggler problem. DTFL introduces a dynamic tier scheduler that uses tier profiling to estimate the expected training time of each client, based on their historical training time, communication speed, and dataset size. The dynamic tier scheduler assigns clients to suitable tiers to minimize the overall training time in each round. We first theoretically prove the convergence properties of DTFL. We then train large models (ResNet-56 and ResNet-110) on popular image datasets (CIFAR-10, CIFAR-100, CINIC-10, and HAM10000) under both IID and non-IID systems. Extensive experimental results show that compared with state-of-the-art FL methods, DTFL can significantly reduce the training time while maintaining model accuracy.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 37 of its 52 references that are in the map. First posted on arXiv 9 Dec 2023.

Builds on

37 of 52 references.

References · 37

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  4. 2019Advances and Open Problems in Federated Learningcited by 3,046
  5. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  6. 2016Deep Learning with Differential Privacycited by 2,728
  7. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,229
  8. 2020Adaptive Federated Optimizationcited by 1,046
  9. 2019Deep Leakage from Gradientscited by 1,021
  10. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 906
  11. 2019Towards Federated Learning at Scale: System Designcited by 832
  12. 2018LEAF: A Benchmark for Federated Settingscited by 792
25 more
  1. 2018Local SGD Converges Fast and Communicates Littlecited by 605
  2. 2021Federated Learning on Non-IID Data Silos: An Experimental Studycited by 568
  3. 2020Federated Learning with Matched Averagingcited by 564
  4. 2018Certified Robustness to Adversarial Examples with Differential Privacycited by 468
  5. 2018Split learning for health: Distributed deep learning without sharing raw patient datacited by 395
  6. 2019FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantizationcited by 356
  7. 2021See through Gradients: Image Batch Recovery via GradInversioncited by 326
  8. 2020SplitFed: When Federated Learning Meets Split Learningcited by 310
  9. 2020FedML: A Research Library and Benchmark for Federated Machine Learningcited by 300
  10. 2018Distributed learning of deep neural network over multiple agentscited by 279
  11. 2018Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learningcited by 278
  12. 2019Training Neural Networks with Local Error Signalscited by 136
  13. 2020TiFL: A Tier-based Federated Learning Systemcited by 96
  14. 2019Decoupled Greedy Learning of CNNscited by 78
  15. 2022Split Learning over Wireless Networks: Parallel Design and Resource Managementcited by 75
  16. 2020NoPeek: Information leakage reduction to share activations in distributed deep learningcited by 66
  17. 2020FedAT: A High-Performance and Communication-Efficient Federated Learning System with Asynchronous Tierscited by 64
  18. 2020Straggler-Resilient Federated Learning: Leveraging the Interplay Between Statistical Accuracy and System Heterogeneitycited by 56
  19. 2020Improving Robustness to Model Inversion Attacks via Mutual Information Regularizationcited by 52
  20. 2018Decoupled Parallel Backpropagation with Convergence Guaranteecited by 39
  21. 2023Privacy and Efficiency of Communications in Federated Split Learningcited by 21
  22. 2021SplitGuard: Detecting and Mitigating Training-Hijacking Attacks in Split Learningcited by 20
  23. 2020Parallel Training of Deep Networks with Local Updatescited by 19
  24. 2022Federated Split BERT for Heterogeneous Text Classificationcited by 11
  25. 2023Secure Vertical Federated Learning Under Unreliable Connectivitycited by 4

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 2.

3 papers.

  1. 2024Communication-Efficient Training Workload Balancing for Decentralized Multi-Agent Learningcited by 2
  2. 2025FedADP: Unified Model Aggregation for Federated Learning with Heterogeneous Model Architecturescited by 2
  3. 2025FedX: Adaptive Model Decomposition and Quantization for IoT Federated Learningcited by 0