mlsanityData to 7 Oct 2026

Every Parameter Matters: Ensuring the Convergence of Federated Learning with Dynamic Heterogeneous Models Reduction

arXiv 2310.08670 ↗First posted 12 Oct 2023cs.LG cs.DCcited by 24

Not on the map. Why?

Abstract (arXiv)

Cross-device Federated Learning (FL) faces significant challenges where low-end clients that could potentially make unique contributions are excluded from training large models due to their resource bottlenecks. Recent research efforts have focused on model-heterogeneous FL, by extracting reduced-size models from the global model and applying them to local clients accordingly. Despite the empirical success, general theoretical guarantees of convergence on this method remain an open question. This paper presents a unifying framework for heterogeneous FL algorithms with online model extraction and provides a general convergence analysis for the first time. In particular, we prove that under certain sufficient conditions and for both IID and non-IID data, these algorithms converge to a stationary point of standard FL for general smooth cost functions. Moreover, we introduce the concept of minimum coverage index, together with model reduction noise, which will determine the convergence of heterogeneous federated learning, and therefore we advocate for a holistic approach that considers both factors to enhance the efficiency of heterogeneous federated learning.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 55 of its 73 references that are in the map. First posted on arXiv 12 Oct 2023.

Builds on

55 of 73 references.

References · 55

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  2. 2016Wide Residual Networkscited by 3,057
  3. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  4. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  5. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,531
  6. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,461
  7. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
  8. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 902
  9. 2019Towards Federated Learning at Scale: System Designcited by 826
  10. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
  11. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 596
  12. 2018Local SGD Converges Fast and Communicates Littlecited by 595
38 more
  1. 2018Rethinking the Value of Network Pruningcited by 595
  2. 2020What is the State of Neural Network Pruning?cited by 578
  3. 2019Rigging the Lottery: Making All Tickets Winnerscited by 491
  4. 2018Adaptive Federated Learning in Resource Constrained Edge Computing Systemscited by 408
  5. 2019Federated Learning in Mobile Edge Networks: A Comprehensive Surveycited by 407
  6. 2020HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clientscited by 334
  7. 2020SplitFed: When Federated Learning Meets Split Learningcited by 308
  8. 2018Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learningcited by 277
  9. 2016Dynamic Network Surgery for Efficient DNNscited by 233
  10. 2017NISP: Pruning Networks using Neuron Importance Score Propagationcited by 228
  11. 2019Model Pruning Enables Efficient Federated Learning on Edge Devicescited by 219
  12. 2021FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropoutcited by 192
  13. 2019Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterizationcited by 187
  14. 2018Expanding the Reach of Federated Learning by Reducing Client Resource Requirementscited by 162
  15. 2017Deep Rewiring: Training very sparse deep networkscited by 136
  16. 2020Dynamic Model Pruning with Feedbackcited by 125
  17. 2018Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithmscited by 106
  18. 2019One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizerscited by 102
  19. 2022FedRolex: Model-Heterogeneous Federated Learning with Rolling Sub-Model Extractioncited by 102
  20. 2019Communication-efficient distributed SGD with Sketchingcited by 81
  21. 2020Adaptive Gradient Sparsification for Efficient Federated Learning: An Online Learning Approachcited by 80
  22. 2018Adaptive Communication Strategies to Achieve the Best Error-Runtime Trade-off in Local-Update SGDcited by 76
  23. 2017Exploring Sparsity in Recurrent Neural Networkscited by 75
  24. 2020Cost-Effective Federated Learning Designcited by 61
  25. 2022A Unified Analysis of Federated Learning with Arbitrary Client Participationcited by 45
  26. 2022Faster Adaptive Federated Learningcited by 35
  27. 2018Graph Oracle Models, Lower Bounds, and Gaps for Parallel Stochastic Optimizationcited by 34
  28. 2021FedHM: Efficient Federated Learning for Heterogeneous Models via Low-rank Factorizationcited by 31
  29. 2022A Multi-agent Reinforcement Learning Approach for Efficient Client Selection in Federated Learningcited by 28
  30. 2021Effective Model Sparsification by Scheduled Grow-and-Prune Methodscited by 25
  31. 2023MAC-PO: Multi-Agent Experience Replay via Collective Priority Optimizationcited by 20
  32. 2022Option-Aware Adversarial Inverse Reinforcement Learning for Robotic Controlcited by 16
  33. 2019Helios: Heterogeneity-Aware Federated Learning with Dynamically Balanced Collaborationcited by 15
  34. 2022Value Functions Factorization with Latent State Information Sharing in Decentralized Multi-Agent Policy Gradientscited by 12
  35. 2022A Bayesian Optimization Framework for Finding Local Optima in Expensive Multi-Modal Functionscited by 12
  36. 2022Federated Select: A Primitive for Communication- and Memory-Efficient Federated Learningcited by 11
  37. 2023Solving a Class of Non-Convex Minimax Optimization in Federated Learningcited by 10
  38. 2023A Versatile Multi-Agent Reinforcement Learning Benchmark for Inventory Managementcited by 9

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 9; 2025: 8; 2026: 5.

24 papers.

Sort:

By area:

  1. 2024FIARSE: Model-Heterogeneous Federated Learning via Importance-Aware Submodel Extractioncited by 21
  2. 2024Auto-Train-Once: Controller Network Guided Automatic Network Pruning from Scratchcited by 20
  3. 2024DapperFL: Domain Adaptive Federated Learning with Model Fusion Pruning for Edge Devicescited by 9
  4. 2025Federated Learning for Cross-Domain Data Privacy: A Distributed Approach to Secure Collaborationcited by 9
  5. 2024Bayesian Optimization through Gaussian Cox Process Models for Spatio-temporal Datacited by 5
  6. 2025MALinZero: Efficient Low-Dimensional Search for Mastering Complex Multi-Agent Planningcited by 5
  7. 2024Unity is Power: Semi-Asynchronous Collaborative Training of Large-Scale Models with Structured Pruning in Resource-Limited Clientscited by 3
  8. 2025FedEL: Federated Elastic Learning for Heterogeneous Devicescited by 3
  9. 2025Research in Collaborative Learning Does Not Serve Cross-Silo Federated Learning in Practicecited by 3
  10. 2024Using Diffusion Models as Generative Replay in Continual Federated Learning -- What will Happen?cited by 2
  11. 2023Federated Learning over Hierarchical Wireless Networks: Training Latency Minimization via Submodel Partitioningcited by 1
  12. 2024Federating Dynamic Models using Early-Exit Architectures for Automatic Speech Recognition on Heterogeneous Clientscited by 1
12 more
  1. 2024FedDTPT: Federated Discrete and Transferable Prompt Tuning for Black-Box Large Language Modelscited by 1
  2. 2025All is Not Lost: LLM Recovery without Checkpointscited by 1
  3. 2025On the Convergence and Stability of Distributed Sub-model Trainingcited by 1
  4. 2023Physics-Enhanced Multi-fidelity Learning for Optical Surface Imprintcited by 0
  5. 2024Adaptive Guidance for Local Training in Heterogeneous Federated Learningcited by 0
  6. 2025A Two-Timescale Approach for Wireless Federated Learning with Parameter Freezing and Power Controlcited by 0
  7. 2025Efficient Asynchronous Federated Evaluation with Strategy Similarity Awareness for Intent-Based Networking in Industrial Internet of Thingscited by 0
  8. 2026FedRD: Reducing Divergences for Generalized Federated Learning via Heterogeneity-aware Parameter Guidancecited by 0
  9. 2026Robust Federated Learning via Byzantine Filtering over Encrypted Updatescited by 0
  10. 2026Representation-Aligned Multi-Scale Personalization for Federated Learningcited by 0
  11. 2026UB-SMoE: Universally Balanced Sparse Mixture-of-Experts for Resource-adaptive Federated Fine-tuning of Foundation Modelscited by 0
  12. 2026From Data Heterogeneity to Convergence: A Data-Centric Review of Federated Learningcited by 0