mlsanityData to 7 Oct 2026

FedRolex: Model-Heterogeneous Federated Learning with Rolling Sub-Model Extraction

arXiv 2212.01548 ↗First posted 3 Dec 2022cs.LG cs.CR cs.CVcited by 102

Not on the map. Why?

Abstract (arXiv)

Most cross-device federated learning (FL) studies focus on the model-homogeneous setting where the global server model and local client models are identical. However, such constraint not only excludes low-end clients who would otherwise make unique contributions to model training but also restrains clients from training large models due to on-device resource bottlenecks. In this work, we propose FedRolex, a partial training (PT)-based approach that enables model-heterogeneous FL and can train a global server model larger than the largest client model. At its core, FedRolex employs a rolling sub-model extraction scheme that allows different parts of the global server model to be evenly trained, which mitigates the client drift induced by the inconsistency between individual client models and server model architectures. We show that FedRolex outperforms state-of-the-art PT-based model-heterogeneous FL methods (e.g. Federated Dropout) and reduces the gap between model-heterogeneous and model-homogeneous FL, especially under the large-model large-dataset regime. In addition, we provide theoretical statistical analysis on its advantage over Federated Dropout and evaluate FedRolex on an emulated real-world device distribution to show that FedRolex can enhance the inclusiveness of FL and boost the performance of low-end devices that would otherwise not benefit from FL. Our code is available at: https://github.com/AIoT-MLSys-Lab/FedRolex

Key facts. Cited by 102 papers in the map (data through 7 Oct 2026). Builds on 26 of its 34 references that are in the map. First posted on arXiv 3 Dec 2022.

Builds on

26 of 34 references.

References · 26

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  6. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  7. 2019Advances and Open Problems in Federated Learningcited by 3,021
  8. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  9. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 678
  10. 2019Federated Learning with Personalization Layerscited by 533
  11. 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 516
  12. 2020HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clientscited by 334
14 more
  1. 2021A Field Guide to Federated Optimizationcited by 312
  2. 2020Attack of the Tails: Yes, You Really Can Backdoor Federated Learningcited by 307
  3. 2021FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropoutcited by 192
  4. 2021Does Knowledge Distillation Really Work?cited by 187
  5. 2016Practical Secure Aggregation for Federated Learning on User-Held Datacited by 178
  6. 2020FedBE: Making Bayesian Model Ensemble Applicable to Federated Learningcited by 176
  7. 2018Expanding the Reach of Federated Learning by Reducing Client Resource Requirementscited by 162
  8. 2019Learning Data Augmentation Strategies for Object Detectioncited by 139
  9. 2020Distillation-Based Semi-Supervised Federated Learning for Communication-Efficient Collaborative Training with Non-IID Private Datacited by 125
  10. 2022Heterogeneous Ensemble Knowledge Transfer for Training Large Models in Federated Learningcited by 81
  11. 2020Siloed Federated Learning for Multi-Centric Histopathology Datasetscited by 61
  12. 2022Scaling Language Model Size in Cross-Device Federated Learningcited by 25
  13. 2021Partial Variable Training for Efficient On-Device Federated Learningcited by 14
  14. 2022Federated Select: A Primitive for Communication- and Memory-Efficient Federated Learningcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 26; 2024: 38; 2025: 26; 2026: 12.

102 papers.

Sort:

By area:

  1. 2023SLoRA: Federated Parameter Efficient Fine-Tuning of Language Modelscited by 92
  2. 2023FedGH: Heterogeneous Federated Learning with Generalized Global Headercited by 56
  3. 2023Federated Learning for Computationally-Constrained Heterogeneous Devices: A Surveycited by 40
  4. 2023pFedLoRA: Model-Heterogeneous Personalized Federated Learning with LoRA Tuningcited by 38
  5. 2023TimelyFL: Heterogeneity-aware Asynchronous Federated Learning with Adaptive Partial Trainingcited by 27
  6. 2023GPT-FL: Generative Pre-trained Model-Assisted Federated Learningcited by 24
  7. 2023Every Parameter Matters: Ensuring the Convergence of Federated Learning with Dynamic Heterogeneous Models Reductioncited by 24
  8. 2023DFRD: Data-Free Robustness Distillation for Heterogeneous Federated Learningcited by 23
  9. 2024pFedMoE: Data-Level Personalization with Mixture of Experts for Model-Heterogeneous Personalized Federated Learningcited by 21
  10. 2024Federated Model Heterogeneous Matryoshka Representation Learningcited by 21
  11. 2024FIARSE: Model-Heterogeneous Federated Learning via Importance-Aware Submodel Extractioncited by 21
  12. 2023A Survey for Federated Learning Evaluations: Goals and Measurescited by 16
38 more
  1. 2023Is Aggregation the Only Choice? Federated Learning via Layer-wise Model Recombinationcited by 13
  2. 2023Multimodal Federated Learning in Healthcare: a Reviewcited by 13
  3. 2024FedP3: Federated Personalized and Privacy-friendly Network Pruning under Model Heterogeneitycited by 13
  4. 2024Advances in Robust Federated Learning: A Survey with Heterogeneity Considerationscited by 12
  5. 2023Agglomerative Federated Learning: Empowering Larger Model Training via End-Edge-Cloud Collaborationcited by 10
  6. 2024Breaking the Memory Wall for Heterogeneous Federated Learning via Progressive Trainingcited by 10
  7. 2023pFedES: Model Heterogeneous Personalized Federated Learning with Feature Extractor Sharingcited by 9
  8. 2024Effective Heterogeneous Federated Learning via Efficient Hypernetwork-based Weight Generationcited by 9
  9. 2024DapperFL: Domain Adaptive Federated Learning with Model Fusion Pruning for Edge Devicescited by 9
  10. 2023Maestro: Uncovering Low-Rank Structures via Trainable Decompositioncited by 8
  11. 2023FedAIoT: A Federated Learning Benchmark for Artificial Intelligence of Thingscited by 8
  12. 2024A Comprehensive Survey of Federated Transfer Learning: Challenges, Methods and Applicationscited by 8
  13. 2023Aggregating Capacity in FL through Successive Layer Training for Computationally-Constrained Devicescited by 7
  14. 2025FedConv: A Learning-on-Model Paradigm for Heterogeneous Federated Clientscited by 7
  15. 2023Towards a Better Theoretical Understanding of Independent Subnetwork Trainingcited by 6
  16. 2023Distributed Personalized Empirical Risk Minimizationcited by 6
  17. 2024Why Go Full? Elevating Federated Learning Through Partial Network Updatescited by 6
  18. 2024Exploring Gradient Subspaces: Addressing and Overcoming LoRA's Limitations in Federated Fine-Tuning of Large Language Modelscited by 6
  19. 2023Explicit Personalization and Local Training: Double Communication Acceleration in Federated Learningcited by 5
  20. 2023Internal Cross-layer Gradients for Extending Homogeneity to Heterogeneity in Federated Learningcited by 5
  21. 2024Towards Diverse Device Heterogeneous Federated Learning via Task Arithmetic Knowledge Integrationcited by 5
  22. 2023Federated Learning Over Images: Vertical Decompositions and Pre-Trained Backbones Are Difficult to Beatcited by 4
  23. 2024DFML: Decentralized Federated Mutual Learningcited by 4
  24. 2024Exploring the Practicality of Federated Learning: A Survey Towards the Communication Perspectivecited by 4
  25. 2024pFedAFM: Adaptive Feature Mixture for Batch-Level Personalization in Heterogeneous Federated Learningcited by 3
  26. 2024Unity is Power: Semi-Asynchronous Collaborative Training of Large-Scale Models with Structured Pruning in Resource-Limited Clientscited by 3
  27. 2025Federated nnU-Net for Privacy-Preserving Medical Image Segmentationcited by 3
  28. 2025Generative Artificial Intelligence for Internet of Things Computing: A Systematic Surveycited by 3
  29. 2025Model Parallelism With Subnetwork Data Parallelismcited by 3
  30. 2025FedEL: Federated Elastic Learning for Heterogeneous Devicescited by 3
  31. 2023Straggler-resilient Federated Learning: Tackling Computation Heterogeneity with Layer-wise Partial Model Training in Mobile Edge Networkcited by 2
  32. 2024FedBRB: An Effective Solution to the Small-to-Large Scenario in Device-Heterogeneity Federated Learningcited by 2
  33. 2024Addressing Heterogeneity in Federated Learning: Challenges and Solutions for a Shared Production Environmentcited by 2
  34. 2024DFDG: Data-Free Dual-Generator Adversarial Distillation for One-Shot Federated Learningcited by 2
  35. 2024FedPeWS: Personalized Warmup via Subnetworks for Enhanced Heterogeneous Federated Learningcited by 2
  36. 2024EFTViT: Efficient Federated Training of Vision Transformers with Masked Images on Resource-Constrained Clientscited by 2
  37. 2024Learnable Sparse Customization in Heterogeneous Edge Computingcited by 2
  38. 2025Collaborative Learning of On-Device Small Model and Cloud-Based Large Model: Advances and Future Directionscited by 2

50 most cited of 102 All 102 (JSON)