mlsanityData to 7 Oct 2026

Hierarchical Federated Learning with Multi-Timescale Gradient Correction

arXiv 2409.18448 ↗First posted 27 Sep 2024cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

While traditional federated learning (FL) typically focuses on a star topology where clients are directly connected to a central server, real-world distributed systems often exhibit hierarchical architectures. Hierarchical FL (HFL) has emerged as a promising solution to bridge this gap, leveraging aggregation points at multiple levels of the system. However, existing algorithms for HFL encounter challenges in dealing with multi-timescale model drift, i.e., model drift occurring across hierarchical levels of data heterogeneity. In this paper, we propose a multi-timescale gradient correction (MTGC) methodology to resolve this issue. Our key idea is to introduce distinct control variables to (i) correct the client gradient towards the group gradient, i.e., to reduce client model drift caused by local updates based on individual datasets, and (ii) correct the group gradient towards the global gradient, i.e., to reduce group model drift caused by FL over clients within the group. We analytically characterize the convergence behavior of MTGC under general non-convex settings, overcoming challenges associated with couplings between correction terms. We show that our convergence bound is immune to the extent of data heterogeneity, confirming the stability of the proposed algorithm against multi-level non-i.i.d. data. Through extensive experiments on various datasets and models, we validate the effectiveness of MTGC in diverse HFL settings. The code for this project is available at hrefhttps://github.com/wenzhifang/MTGChttps://github.com/wenzhifang/MTGC.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 44 of its 58 references that are in the map. First posted on arXiv 27 Sep 2024.

Builds on

44 of 58 references.

References · 44

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  2. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  3. 2019Advances and Open Problems in Federated Learningcited by 3,021
  4. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  5. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,531
  6. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,461
  7. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
  8. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
  9. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 897
  10. 2018Local SGD Converges Fast and Communicates Littlecited by 595
  11. 2021Federated Learning Based on Dynamic Regularizationcited by 555
  12. 2020A Unified Theory of Decentralized SGD with Changing Topology and Local Updatescited by 361
32 more
  1. 2021A Field Guide to Federated Optimizationcited by 312
  2. 2018Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learningcited by 277
  3. 2019Tighter Theory for Local SGD on Identical and Heterogeneous Datacited by 266
  4. 2017EMNIST: an extension of MNIST to handwritten letterscited by 212
  5. 2019Client-Edge-Cloud Hierarchical Federated Learningcited by 202
  6. 2019On the Convergence of Local Descent Methods in Federated Learningcited by 150
  7. 2016Achieving Geometric Convergence for Distributed Optimization over Time-Varying Graphscited by 149
  8. 2020FedPD: A Federated Learning Framework with Optimal Rates and Adaptivity to Non-IID Datacited by 143
  9. 2022ProxSkip: Yes! Local Gradient Steps Provably Lead to Communication Acceleration! Finally!cited by 134
  10. 2020FedSplit: An algorithmic framework for fast federated optimizationcited by 108
  11. 2016NEXT: In-Network Nonconvex Optimizationcited by 107
  12. 2019Variance Reduced Local SGD with Lower Communication Complexitycited by 80
  13. 2024FedBiOT: LLM Local Fine-tuning in Federated Learning without Full Modelcited by 73
  14. 2022Communication-Efficient Stochastic Zeroth-Order Optimization for Federated Learningcited by 59
  15. 2021Hierarchical Federated Learning with Quantization: Convergence Analysis and System Designcited by 47
  16. 2020Multi-Stage Hybrid Federated Learning over Large-Scale D2D-Enabled Fog Networkscited by 42
  17. 2022Variance Reduced ProxSkip: Algorithm, Theory and Application to Federated Learningcited by 33
  18. 2023Decentralized Gradient Tracking with Local Stepscited by 32
  19. 2022On the Convergence of Clustered Federated Learningcited by 31
  20. 2020Demystifying Why Local Aggregation Helps: Convergence Analysis of Hierarchical SGDcited by 26
  21. 2023Local Exact-Diffusion for Decentralized Optimization and Learningcited by 24
  22. 2018Distributed Nonconvex Constrained Optimization over Time-Varying Digraphscited by 23
  23. 2023Optimizing the Collaboration Structure in Cross-Silo Federated Learningcited by 19
  24. 2020Distributed Algorithms for Composite Optimization: Unified Framework and Convergence Analysiscited by 15
  25. 2021H-FL: A Hierarchical Communication-Efficient and Privacy-Protected Architecture for Federated Learningcited by 14
  26. 2022Anchor Sampling for Federated Learning with Partial Client Participationcited by 14
  27. 2018Achieving Linear Convergence in Distributed Asynchronous Multi-agent Optimizationcited by 11
  28. 2024Federated Optimization with Doubly Regularized Drift Correctioncited by 10
  29. 2021From Deterioration to Acceleration: A Calibration Approach to Rehabilitating Step Asynchronism in Federated Optimizationcited by 8
  30. 2021FedFog: Network-Aware Optimization of Federated Learning over Wireless Fog-Cloud Systemscited by 6
  31. 2023Taming Subnet-Drift in D2D-Enabled Fog Learning: A Hierarchical Gradient Tracking Approachcited by 5
  32. 2023Communication-Efficient Federated Optimization over Semi-Decentralized Networkscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 2.

7 papers.

  1. 2025Federated Sketching LoRA: A Flexible Framework for Heterogeneous Collaborative Fine-Tuning of LLMscited by 2
  2. 2025Gradient Correction in Federated Learning with Adaptive Optimizationcited by 1
  3. 2025Adaptive Federated LoRA in Heterogeneous Wireless Networks with Independent Samplingcited by 1
  4. 2025HFedATM: Hierarchical Federated Domain Generalization via Optimal Transport and Regularized Mean Aggregationcited by 0
  5. 2025SOFA-FL: Self-Organizing Hierarchical Federated Learning with Adaptive Clustered Data Sharingcited by 0
  6. 2026ELSA: Efficient LLM-Centric Split Aggregation for Privacy-Aware Hierarchical Federated Learning over the Network Edgecited by 0
  7. 2026Mitigating Heterogeneity-Induced Drift in Hierarchical Sign-Based Federated Learningcited by 0