mlsanityData to 14 Oct 2026

Federated Fine-Tuning of Sparsely-Activated Large Language Models on Resource-Constrained Devices

arXiv 2508.19078 ↗First posted 26 Aug 2025cs.DC cs.AIcited by 1

On the map: Mixture-of-experts → Federated mixture-of-experts

Abstract (arXiv)

Federated fine-tuning of Mixture-of-Experts (MoE)-based large language models (LLMs) is challenging due to their massive computational requirements and the resource constraints of participants. Existing working attempts to fill this gap through model quantization, computation offloading, or expert pruning. However, they cannot achieve desired performance due to impractical system assumptions and a lack of consideration for MoE-specific characteristics. In this paper, we propose FLUX, a system designed to enable federated fine-tuning of MoE-based LLMs across participants with constrained computing resources (e.g., consumer-grade GPUs), aiming to minimize time-to-accuracy. FLUX introduces three key innovations: (1) quantization-based local profiling to estimate expert activation with minimal overhead, (2) adaptive layer-aware expert merging to reduce resource consumption while preserving accuracy, and (3) dynamic expert role assignment using an exploration-exploitation strategy to balance tuning and non-tuning experts. Extensive experiments on LLaMA-MoE and DeepSeek-MoE with multiple benchmark datasets demonstrate that FLUX significantly outperforms existing methods, achieving up to 4.75X speedup in time-to-accuracy.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Federated mixture-of-experts, one of 12 directions in Mixture-of-experts. Builds on 54 of its 76 references that are in the map. First posted on arXiv 26 Aug 2025.

Builds on

54 of 76 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024FedMoE: Personalized Federated Learning via Heterogeneous Mixture of Experts21 Aug
  2. 2020PFL-MoE: Personalized Federated Learning Based on Mixture of Experts31 Dec

Other references · 52

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  7. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  8. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  9. 2024Mixtral of Expertscited by 1,823
  10. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  11. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  12. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
38 more
  1. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  2. 2021Scaling Vision with Sparse Mixture of Expertscited by 747
  3. 2019A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protectioncited by 374
  4. 2018Not All Samples Are Created Equal: Deep Learning with Importance Samplingcited by 328
  5. 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuningcited by 150
  6. 2023FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learningcited by 149
  7. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Modelscited by 146
  8. 2021Scaling Deep Contrastive Learning Batch Size under Memory Limited Setupcited by 127
  9. 2024LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-trainingcited by 123
  10. 2023OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Modelscited by 114
  11. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policycited by 108
  12. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inferencecited by 104
  13. 2023Fast Inference of Mixture-of-Experts Language Models with Offloadingcited by 91
  14. 2021FedNLP: Benchmarking Federated Learning Methods for Natural Language Processing Taskscited by 85
  15. 2022NVIDIA FLARE: Federated Learning from Simulation to Real-Worldcited by 80
  16. 2024FedBiOT: LLM Local Fine-tuning in Federated Learning without Full Modelcited by 74
  17. 2022Gradients without Backpropagationcited by 73
  18. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUscited by 73
  19. 2023EdgeMoE: Empowering Sparse Large Language Models on Mobile Devicescited by 69
  20. 2023A Hybrid Tensor-Expert-Data Parallelism Approach to Optimize Mixture-of-Experts Trainingcited by 47
  21. 2020Dynamic Tensor Rematerializationcited by 46
  22. 2023QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Modelscited by 42
  23. 2023Emergent Modularity in Pre-trained Transformerscited by 37
  24. 2023Merging Experts into One: Improving Computational Efficiency of Mixture of Expertscited by 37
  25. 2021FedGraph: Federated Graph Learning with Intelligent Samplingcited by 34
  26. 2023Mixture of Quantized Experts (MoQE): Complementary Effect of Low-bit Quantization and Robustnesscited by 34
  27. 2023SwapMoE: Serving Off-the-shelf MoE-based Large Language Models with Tunable Memory Budgetcited by 27
  28. 2024A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Expertscited by 21
  29. 2024Let the Expert Stick to His Last: Expert-Specialized Fine-Tuning for Sparse Architectural Large Language Modelscited by 21
  30. 2022Federated Few-Shot Learning for Mobile NLPcited by 15
  31. 2023BAFFLE: A Baseline of Backpropagation-Free Federated Learningcited by 12
  32. 2024Gold Panning in Vocabulary: An Adaptive Method for Vocabulary Expansion of Domain-Specific LLMscited by 12
  33. 2024Personalized Federated Fine-Tuning for LLMs via Data-Driven Heterogeneous Model Architecturescited by 12
  34. 2025FlowerTune: A Cross-Domain Benchmark for Federated Fine-Tuning of Large Language Modelscited by 11
  35. 2024Federated Data-Efficient Instruction Tuning for Large Language Modelscited by 10
  36. 2024Quantized Side Tuning: Fast and Memory-Efficient Tuning of Quantized Large Language Modelscited by 9
  37. 2024PERFT: Parameter-Efficient Routed Fine-Tuning for Mixture-of-Expert Modelcited by 9
  38. 2024Llama 3 Meets MoE: Efficient Upcyclingcited by 6

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Fine-Tuning a 3B-Parameter LLM on a Smartphone: Characterizing Sustained Trainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Task-Aware Federated Fine-Tuning for MoE-based Large Language Models
  2. 2026Conflict-Aware Federated Fine-Tuning of Large Language Models with Mixture-of-Experts
  3. 2026HFedMoE: Resource-aware Heterogeneous Federated Learning with Mixture-of-Experts
  4. 2024Understanding the Performance and Estimating the Cost of LLM Fine-Tuning
  5. 2025Mixture of Experts in Large Language Models
  6. 2026From Experts to Sub-experts: Fine-grained Parameter-Efficient Fine-Tuning for MoE LLMs
  7. 2025Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloading
  8. 2026Scaling Multi-Node Mixture-of-Experts Inference Using Expert Activation Patterns