mlsanityData to 14 Oct 2026

UB-SMoE: Universally Balanced Sparse Mixture-of-Experts for Resource-adaptive Federated Fine-tuning of Foundation Models

arXiv 2605.16690 ↗First posted 15 May 2026cs.LGcited by 0

On the map: Mixture-of-experts → Federated mixture-of-experts

Abstract (arXiv)

Heterogeneous LoRA-rank methods address system heterogeneity in federated fine-tuning of foundation models by assigning client-specific ranks based on computational capabilities. However, these methods achieve only marginal computational savings, as dense feed-forward computations dominate. Sparse Mixture-of-Experts (SMoE) provides a promising alternative through conditional computation, yet we identify that its naive application to heterogeneous federated settings introduces two critical discordances: (i) expert utilization imbalance and (ii) non-differentiability of Top-K routing. Our convergence analysis demonstrates that these discordances lead to degraded convergence, particularly for resource-constrained clients. To address these challenges, we propose Universally Balanced Sparse Mixture-of-Experts (UB-SMoE), which introduces Dynamic Modulated Routing (DMR) to rebalance expert utilization, and Universal Pseudo-Gradient (PG) to reconstruct learning signals for non-activated experts. These mechanisms form a self-reinforcing cycle that maintains expert viability across heterogeneous clients. Experiments on benchmarks show that UB-SMoE achieves up to 45.0% computational reduction on low-resource clients while improving their performance by 8.7 × compared to existing heterogeneous LoRA-rank methods.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Federated mixture-of-experts, one of 12 directions in Mixture-of-experts. Builds on 58 of its 67 references that are in the map. First posted on arXiv 15 May 2026.

Builds on

58 of 67 references.

References · 58

  1. 2020Scaling Laws for Neural Language Modelscited by 6,586
  2. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  3. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  6. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  7. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  8. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  9. 2024Mixtral of Expertscited by 1,823
  10. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  11. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  12. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,229
38 more
  1. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,150
  2. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  3. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  4. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 851
  5. 2024OLMo: Accelerating the Science of Language Modelscited by 613
  6. 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 539
  7. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 399
  8. 2019Bayesian Nonparametric Federated Learning of Neural Networkscited by 387
  9. 2020A Unified Theory of Decentralized SGD with Changing Topology and Local Updatescited by 367
  10. 2023LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 324
  11. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
  12. 2019SGD: General Analysis and Improved Ratescited by 207
  13. 2024Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Expertscited by 205
  14. 2023Towards Building the Federated GPT: Federated Instruction Tuningcited by 185
  15. 2024Improving LoRA in Privacy-preserving Federated Learningcited by 161
  16. 2024FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptationscited by 146
  17. 2024Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Modelscited by 116
  18. 2023Stochastic Controlled Averaging for Federated Learning with Communication Compressioncited by 101
  19. 2024Federated Fine-tuning of Large Language Models under Heterogeneous Tasks and Client Resourcescited by 93
  20. 2020On the Convergence of SGD with Biased Gradientscited by 83
  21. 2023FedDAT: An Approach for Foundation Model Finetuning in Multi-Modal Heterogeneous Federated Learningcited by 60
  22. 2023Efficient Model Personalization in Federated Learning via Client-Specific Prompt Generationcited by 54
  23. 2024ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routingcited by 54
  24. 2025Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Modelscited by 54
  25. 2020Biased Stochastic First-Order Methods for Conditional Stochastic Optimization and Applications in Meta Learningcited by 47
  26. 2019Non-asymptotic Analysis of Biased Stochastic Approximation Schemecited by 45
  27. 2023A Guide Through the Zoo of Biased SGDcited by 40
  28. 2023Federated Learning for Computationally-Constrained Heterogeneous Devices: A Surveycited by 40
  29. 2023Every Parameter Matters: Ensuring the Convergence of Federated Learning with Dynamic Heterogeneous Models Reductioncited by 24
  30. 2023Convergence Analysis of Sequential Federated Learning on Heterogeneous Datacited by 23
  31. 2024Federated Learning from Vision-Language Foundation Models: Theoretical Analysis and Methodcited by 22
  32. 2024Initialization using Update Approximation is a Silver Bullet for Extremely Efficient Low-Rank Fine-Tuningcited by 22
  33. 2024Asynchronous Federated Reinforcement Learning with Policy Gradient Updates: Algorithm Design and Convergence Analysiscited by 21
  34. 2024FedEx-LoRA: Exact Aggregation for Federated and Efficient Fine-Tuning of Foundation Modelscited by 18
  35. 2023Sparse Backpropagation for MoE Trainingcited by 17
  36. 2024Convergence Analysis of Split Federated Learning on Heterogeneous Datacited by 17
  37. 2024GRIN: GRadient-INformed MoEcited by 17
  38. 2025A Survey on Parameter-Efficient Fine-Tuning for Foundation Models in Federated Learningcited by 17

50 most cited of 58 All 58 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025FLAME: Towards Federated Fine-Tuning Large Language Models Through Adaptive SMoE
  2. 2026HFedMoE: Resource-aware Heterogeneous Federated Learning with Mixture-of-Experts
  3. 2025FFT-MoE: Efficient Federated Fine-Tuning for Foundation Models via Large-scale Sparse MoE under Heterogeneous Edge
  4. 2026Federated Foundation Models Fine-Tuning with Heterogeneous Compressed Clients
  5. 2024FedMoE-DA: Federated Mixture of Experts via Domain Aware Fine-grained Aggregation
  6. 2026FedWeave: Rethinking the Unit of Specialization in Heterogeneous Federated MoE-LoRA
  7. 2025Adaptive LoRA Experts Allocation and Selection for Federated Fine-Tuning
  8. 2025Selective Sinkhorn Routing for Improved Sparse Mixture of Experts