mlsanityData to 14 Oct 2026

FedHM: Efficient Federated Learning for Heterogeneous Models via Low-rank Factorization

arXiv 2111.14655 ↗First posted 29 Nov 2021cs.LG cs.AI cs.DCcited by 31

Not on the map. Why?

Abstract (arXiv)

One underlying assumption of recent federated learning (FL) paradigms is that all local models usually share the same network architecture and size, which becomes impractical for devices with different hardware resources. A scalable federated learning framework should address the heterogeneity that clients have different computing capacities and communication capabilities. To this end, this paper proposes FedHM, a novel heterogeneous federated model compression framework, distributing the heterogeneous low-rank models to clients and then aggregating them into a full-rank model. Our solution enables the training of heterogeneous models with varying computational complexities and aggregates them into a single global model. Furthermore, FedHM significantly reduces the communication cost by using low-rank models. Extensive experimental results demonstrate that FedHM is superior in the performance and robustness of models of different sizes, compared with state-of-the-art heterogeneous FL methods under various FL settings. Additionally, the convergence guarantee of FL for heterogeneous devices is first theoretically analyzed.

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). Builds on 36 of its 49 references that are in the map. First posted on arXiv 29 Nov 2021.

Builds on

36 of 49 references.

References · 36

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  2. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  3. 2019Advances and Open Problems in Federated Learningcited by 3,046
  4. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,538
  5. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,475
  6. 2020Adaptive Federated Optimizationcited by 1,046
  7. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 906
  8. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 851
  9. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 787
  10. 2021Model-Contrastive Federated Learningcited by 766
  11. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 682
  12. 2020Federated Learning with Matched Averagingcited by 564
24 more
  1. 2021Federated Learning Based on Dynamic Regularizationcited by 562
  2. 2019FedMD: Heterogenous Federated Learning via Model Distillationcited by 459
  3. 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learningcited by 383
  4. 2020HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clientscited by 337
  5. 2020Think Locally, Act Globally: Federated Learning with Local and Global Representationscited by 325
  6. 2021A Field Guide to Federated Optimizationcited by 315
  7. 2018Slimmable Neural Networkscited by 305
  8. 2021FedDG: Federated Domain Generalization on Medical Image Segmentation via Episodic Learning in Continuous Frequency Spacecited by 287
  9. 2018Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learningcited by 278
  10. 2019Universally Slimmable Networks and Improved Training Techniquescited by 221
  11. 2014Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decompositioncited by 203
  12. 2020FetchSGD: Communication-Efficient Federated Learning with Sketchingcited by 179
  13. 2020Self-Distillation Amplifies Regularization in Hilbert Spacecited by 173
  14. 2021UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuningcited by 162
  15. 2018Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysiscited by 156
  16. 2020A Survey of Privacy Attacks in Machine Learningcited by 146
  17. 2021FedPara: Low-Rank Hadamard Product for Communication-Efficient Federated Learningcited by 141
  18. 2021Dynamic Slimmable Networkcited by 83
  19. 2020Stable Low-rank Tensor Decomposition for Compression of Convolutional Neural Networkcited by 74
  20. 2021Pufferfish: Communication-efficient Models At No Extra Costcited by 61
  21. 2021Initialization and Regularization of Factorized Neural Layerscited by 58
  22. 2020Optimal Gradient Compression for Distributed and Federated Learningcited by 39
  23. 2021FedAdapt: Adaptive Offloading for IoT Devices in Federated Learningcited by 26
  24. 2021Communication-Efficient Federated Learning with Dual-Side Low-Rank Compressioncited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 4; 2023: 5; 2024: 12; 2025: 8; 2026: 2.

31 papers.

Sort:

By area:

  1. 2022A Survey on Heterogeneous Federated Learningcited by 42
  2. 2023Federated Learning for Computationally-Constrained Heterogeneous Devices: A Surveycited by 40
  3. 2023Cuttlefish: Low-Rank Model Training without All the Tuningcited by 25
  4. 2023Every Parameter Matters: Ensuring the Convergence of Federated Learning with Dynamic Heterogeneous Models Reductioncited by 24
  5. 2024Advances in Robust Federated Learning: A Survey with Heterogeneity Considerationscited by 12
  6. 2024Recurrent Early Exits for Federated Learning with Heterogeneous Clientscited by 12
  7. 2024Effective Heterogeneous Federated Learning via Efficient Hypernetwork-based Weight Generationcited by 9
  8. 2023Aggregating Capacity in FL through Successive Layer Training for Computationally-Constrained Devicescited by 7
  9. 2024Federated Dynamical Low-Rank Training with Global Loss Convergence Guaranteescited by 7
  10. 2024Communication-Efficient Federated Low-Rank Update Algorithm and its Connection to Implicit Regularizationcited by 6
  11. 2024DeTrigger: A Gradient-Centric Approach to Backdoor Attack Mitigation in Federated Learningcited by 5
  12. 2025The Panaceas for Improving Low-Rank Decomposition in Communication-Efficient Federated Learningcited by 5
19 more
  1. 2022On the Convergence of Heterogeneous Federated Learning with Arbitrary Adaptive Online Model Pruningcited by 4
  2. 2022FedorAS: Federated Architecture Search under system heterogeneitycited by 4
  3. 2024Embracing Federated Learning: Enabling Weak Client Participation via Partial Model Trainingcited by 3
  4. 2022Feature Correlation-guided Knowledge Transfer for Federated Self-supervised Learningcited by 1
  5. 2024Efficient Federated Finetuning of Tiny Transformers with Resource-Constrained Devicescited by 1
  6. 2025Enabling Weak Client Participation via On-device Knowledge Distillation in Heterogeneous Federated Learningcited by 1
  7. 2025Communication-Efficient Personalized Distributed Learning with Data and Node Heterogeneitycited by 1
  8. 2025On the Convergence and Stability of Distributed Sub-model Trainingcited by 1
  9. 2023Heterogeneous federated collaborative filtering using FAIR: Federated Averaging in Random Subspacescited by 0
  10. 2024Federated Learning: A Stochastic Approximation Approachcited by 0
  11. 2024MSfusion: A Dynamic Model Splitting Approach for Resource-Constrained Machines to Collaboratively Train Larger Modelscited by 0
  12. 2024On Sampling Strategies for Spectral Model Shardingcited by 0
  13. 2024Photon: Federated LLM Pre-Trainingcited by 0
  14. 2025Moss: Proxy Model-based Full-Weight Aggregation in Federated Learning with Heterogeneous Modelscited by 0
  15. 2025QA-HFL: Quality-Aware Hierarchical Federated Learning for Resource-Constrained Mobile Devices with Heterogeneous Image Qualitycited by 0
  16. 2025FLAME: Towards Federated Fine-Tuning Large Language Models Through Adaptive SMoEcited by 0
  17. 2025DART: A Server-side Plug-in for Resource-efficient Robust Federated Learningcited by 0
  18. 2026Semantic-based Distributed Learning for Diverse and Discriminative Representationscited by 0
  19. 2026SpeechLLM Meets Federated Learning for End-to-End ASR: English and Italian Case Studiescited by 0