FLAME: Towards Federated Fine-Tuning Large Language Models Through Adaptive SMoE
On the map: Mixture-of-experts → Federated mixture-of-experts
Abstract (arXiv)
Existing resource-adaptive LoRA federated fine-tuning methods enable clients to fine-tune models using compressed versions of global LoRA matrices, in order to accommodate various compute resources across clients. This compression requirement will lead to suboptimal performance due to information loss. To address this, we propose FLAME, a novel federated learning framework based on the Sparse Mixture-of-Experts (SMoE) architecture. Unlike prior approaches, FLAME retains full (uncompressed) global LoRA matrices and achieves client-side adaptability by varying the number of activated experts per client. However, incorporating SMoE into federated learning introduces unique challenges, specifically, the mismatch in output magnitude from partial expert activation and the imbalance in expert training quality across clients. FLAME tackles these challenges through a lightweight rescaling mechanism and an activation-aware aggregation scheme. Empirical results across diverse computational settings demonstrate that FLAME consistently outperforms existing methods, providing a robust and effective solution for resource-adaptive federated learning.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Federated mixture-of-experts, one of 12 directions in Mixture-of-experts. Builds on 37 of its 52 references that are in the map. First posted on arXiv 19 Jun 2025.
Builds on
37 of 52 references.
Closest ancestors · 2
Same direction, newest first
- 2025FLEx: Personalized Federated Learning for Mixture-of-Experts LLMs via Expert Grafting1 Jun
- 2025PM-MOE: Mixture of Experts on Private Model Parameters for Personalized Federated Learning1 Feb
Other references · 35
- 2023GPT-4 Technical Reportcited by 18,132
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
- 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
- 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
- 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
- 2024OLMo: Accelerating the Science of Language Modelscited by 599
- 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
- 2024A Survey on Mixture of Experts in Large Language Modelscited by 233
- 2022Language model compression with weighted low-rank factorizationcited by 192
- 2023Towards Building the Federated GPT: Federated Instruction Tuningcited by 184
- 2024Improving LoRA in Privacy-preserving Federated Learningcited by 158
- 2023FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learningcited by 146
23 more
- 2024FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptationscited by 144
- 2022PromptFL: Let Federated Participants Cooperatively Learn Prompts Instead of Models -- Federated Learning in Age of Foundation Modelcited by 133
- 2024Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Modelscited by 114
- 2024LLaVA-MoLE: Sparse Mixture of LoRA Experts for Mitigating Data Conflicts in Instruction Finetuning MLLMscited by 84
- 2024Selective Aggregation for Low-Rank Adaptation in Federated Learningcited by 80
- 2022When Federated Learning Meets Pre-trained Language Models' Parameter-Efficient Tuning Methodscited by 74
- 2023Federated Learning of Large Language Models with Parameter-Efficient Prompt Tuning and Adaptive Optimizationcited by 65
- 2024The Ultimate Guide to Fine-Tuning LLMs from Basics to Breakthroughs: An Exhaustive Review of Technologies, Research, Best Practices, Applied Research Challenges and Opportunitiescited by 63
- 2019Survey of Dropout Methods for Deep Neural Networkscited by 44
- 2023QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Modelscited by 42
- 2021FedHM: Efficient Federated Learning for Heterogeneous Models via Low-rank Factorizationcited by 31
- 2024FeDeRA:Efficient Fine-tuning of Language Models in Federated Learning Leveraging Weight Decompositioncited by 26
- 2023FedBPT: Efficient Federated Black-box Prompt Tuning for Large Language Modelscited by 25
- 2022FedAdapter: Efficient Federated Learning for Modern NLPcited by 24
- 2025A Survey on Federated Fine-tuning of Large Language Modelscited by 21
- 2023Federated Fine-Tuning of LLMs on the Very Edge: The Good, the Bad, the Uglycited by 17
- 2025Robust Federated Finetuning of LLMs via Alternating Optimization of LoRAcited by 17
- 2022Numerical Optimizations for Weighted Low-rank Estimation on Language Modelcited by 14
- 2024Communication-Efficient and Tensorized Federated Fine-Tuning of Large Language Modelscited by 12
- 2024C2A: Client-Customized Adaptation for Parameter-Efficient Federated Learningcited by 8
- 2023Improved Training of Mixture-of-Experts Language GANscited by 7
- 2025AFLoRA: Adaptive Federated Fine-Tuning of Large Language Models with Resource-Aware Low-Rank Adaptioncited by 5
- 2025FLoRIST: Singular Value Thresholding for Efficient and Accurate Federated Fine-Tuning of Large Language Modelscited by 4
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Adaptive LoRA Experts Allocation and Selection for Federated Fine-Tuning
- 2026UB-SMoE: Universally Balanced Sparse Mixture-of-Experts for Resource-adaptive Federated Fine-tuning of Foundation Models
- 2025FedALT: Federated Fine-Tuning through Adaptive Local Training with Rest-of-World LoRA
- 2024FedEx-LoRA: Exact Aggregation for Federated and Efficient Fine-Tuning of Foundation Models
- 2026Conflict-Aware Federated Fine-Tuning of Large Language Models with Mixture-of-Experts
- 2026FLoRG: Federated Fine-tuning with Low-rank Gram Matrices and Procrustes Alignment
- 2026CFLoRA: Federated Fine-tuning of LLMs with Complementary Factors for Error-free Aggregation
- 2026HFedMoE: Resource-aware Heterogeneous Federated Learning with Mixture-of-Experts