mlsanityData to 7 Oct 2026

Effective Heterogeneous Federated Learning via Efficient Hypernetwork-based Weight Generation

arXiv 2407.03086 ↗First posted 3 Jul 2024cs.LG cs.AI cs.DCcited by 9

Not on the map. Why?

Abstract (arXiv)

While federated learning leverages distributed client resources, it faces challenges due to heterogeneous client capabilities. This necessitates allocating models suited to clients' resources and careful parameter aggregation to accommodate this heterogeneity. We propose HypeMeFed, a novel federated learning framework for supporting client heterogeneity by combining a multi-exit network architecture with hypernetwork-based model weight generation. This approach aligns the feature spaces of heterogeneous model layers and resolves per-layer information disparity during weight aggregation. To practically realize HypeMeFed, we also propose a low-rank factorization approach to minimize computation and memory overhead associated with hypernetworks. Our evaluations on a real-world heterogeneous device testbed indicate that system enhances accuracy by 5.12% over FedAvg, reduces the hypernetwork memory requirements by 98.22%, and accelerates its operations by 1.86x compared to a naive hypernetwork approach. These results demonstrate HypeMeFed's effectiveness in leveraging and engaging heterogeneous clients for federated learning.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 42 of its 77 references that are in the map. First posted on arXiv 3 Jul 2024.

Builds on

42 of 77 references.

References · 42

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  6. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  7. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  8. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  9. 2020Implicit Neural Representations with Periodic Activation Functionscited by 2,214
  10. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  11. 2016HyperNetworkscited by 1,003
  12. 2019Towards Federated Learning at Scale: System Designcited by 826
30 more
  1. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 697
  2. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 678
  3. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 591
  4. 2019FedMD: Heterogenous Federated Learning via Model Distillationcited by 454
  5. 2018Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfacescited by 376
  6. 2020HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clientscited by 334
  7. 2016Dynamic Filter Networkscited by 318
  8. 2021No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Datacited by 241
  9. 2021Personalized Federated Learning using Hypernetworkscited by 210
  10. 2021FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropoutcited by 192
  11. 2022On-Device Training Under 256KB Memorycited by 158
  12. 2022FedRolex: Model-Heterogeneous Federated Learning with Rolling Sub-Model Extractioncited by 102
  13. 2023A Brief Review of Hypernetworks in Deep Learningcited by 102
  14. 2020Gradient Flow in Sparse Neural Networks and How Lottery Tickets Wincited by 77
  15. 2021Adaptive Inference through Early-Exit Networks: Design, Challenges and Directionscited by 74
  16. 2019Hypernetwork functional image representationcited by 65
  17. 2020LotteryFL: Personalized and Communication-Efficient Federated Learning with Lottery Ticket Hypothesis on Non-IID Datasetscited by 60
  18. 2016UniMiB SHAR: a new dataset for human activity recognition using acceleration data from smartphonescited by 54
  19. 2019Deep Meta Functionals for Shape Representationcited by 53
  20. 2022FedBalancer: Data and Pace Control for Efficient Federated Learning on Heterogeneous Clientscited by 46
  21. 2022Orchestra: Unsupervised Federated Learning via Globally Consistent Clusteringcited by 45
  22. 2021FedHM: Efficient Federated Learning for Heterogeneous Models via Low-rank Factorizationcited by 31
  23. 2022HINT: Hypernetwork Instruction Tuning for Efficient Zero- & Few-Shot Generalisationcited by 28
  24. 2021It's always personal: Using Early Exits for Efficient On-Device CNN Personalisationcited by 18
  25. 2022Federated Learning with Heterogeneous Architectures using Graph HyperNetworkscited by 17
  26. 2024MESEN: Exploit Multimodal Data to Design Unimodal Human Activity Recognition with Few Labelscited by 11
  27. 2024Recurrent Early Exits for Federated Learning with Heterogeneous Clientscited by 11
  28. 2022FLAME: Federated Learning Across Multi-device Environmentscited by 8
  29. 2024Improving Local Training in Federated Learning via Temperature Scalingcited by 5
  30. 2023Federated Learning Over Images: Vertical Decompositions and Pre-Trained Backbones Are Difficult to Beatcited by 4

Built on by

Cited by year

Citing papers per year: 2024: 3; 2025: 6.

9 papers.

  1. 2025One-shot Federated Learning Methods: A Practical Guidecited by 6
  2. 2024DeTrigger: A Gradient-Centric Approach to Backdoor Attack Mitigation in Federated Learningcited by 5
  3. 2024FedAli: Personalized Federated Learning Alignment with Prototype Layers for Generalized Mobile Servicescited by 2
  4. 2024Tazza: Shuffling Neural Network Parameters for Secure and Private Federated Learningcited by 2
  5. 2025Deploying Models to Non-participating Clients in Federated Learning without Fine-tuning: A Hypernetwork-based Approachcited by 2
  6. 2025Individualised Treatment Effects Estimation with Composite Treatments and Composite Outcomescited by 1
  7. 2025Hypernetworks for Model-Heterogeneous Personalized Federated Learningcited by 1
  8. 2025FedPalm: A General Federated Learning Framework for Closed- and Open-Set Palmprint Verificationcited by 0
  9. 2025Toward Storage-Aware Learning with Compressed Data An Empirical Exploratory Study on JPEGcited by 0