mlsanityData to 14 Oct 2026

FedMerge: Federated Personalization via Model Merging

arXiv 2504.06768 ↗First posted 9 Apr 2025cs.LGcited by 4

On the map: Mixture-of-experts → Federated mixture-of-experts

Abstract (arXiv)

One global model in federated learning (FL) might not be sufficient to serve many clients with non-IID tasks and distributions. While there has been advances in FL to train multiple global models for better personalization, they only provide limited choices to clients so local finetuning is still indispensable. In this paper, we propose a novel ``FedMerge'' approach that can create a personalized model per client by simply merging multiple global models with automatically optimized and customized weights. In FedMerge, a few global models can serve many non-IID clients, even without further local finetuning. We formulate this problem as a joint optimization of global models and the merging weights for each client. Unlike existing FL approaches where the server broadcasts one or multiple global models to all clients, the server only needs to send a customized, merged model to each client. Moreover, instead of periodically interrupting the local training and re-initializing it to a global model, the merged model aligns better with each client's task and data distribution, smoothening the local-global gap between consecutive rounds caused by client drift. We evaluate FedMerge on three different non-IID settings applied to different domains with diverse tasks and data types, in which FedMerge consistently outperforms existing FL approaches, including clustering-based and mixture-of-experts (MoE) based methods.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Federated mixture-of-experts, one of 12 directions in Mixture-of-experts. Builds on 47 of its 61 references that are in the map. First posted on arXiv 9 Apr 2025.

Builds on

47 of 61 references.

References · 47

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  4. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  5. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  6. 2014How transferable are features in deep neural networks?cited by 1,225
  7. 2022Editing Models with Task Arithmeticcited by 1,120
  8. 2017Deeper, Broader and Artier Domain Generalizationcited by 986
  9. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  10. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 851
  11. 2020Ditto: Fair and Robust Federated Learning Through Personalizationcited by 672
  12. 2021Merging Models with Fisher-Weighted Averagingcited by 621
35 more
  1. 2020Personalized Federated Learning with Moreau Envelopescited by 618
  2. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 578
  3. 2021Exploiting Shared Representations for Personalized Federated Learningcited by 541
  4. 2020An Efficient Framework for Clustered Federated Learningcited by 511
  5. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 364
  6. 2023Character-LLM: A Trainable Agent for Role-Playingcited by 339
  7. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 254
  8. 2021Federated Multi-Task Learning under a Mixture of Distributionscited by 192
  9. 2023Towards Building the Federated GPT: Federated Instruction Tuningcited by 185
  10. 2024Improving LoRA in Privacy-preserving Federated Learningcited by 161
  11. 2023FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learningcited by 149
  12. 2023Routing to the Expert: Efficient Reward-guided Ensemble of Large Language Modelscited by 145
  13. 2024OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learningcited by 128
  14. 2024Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Modelscited by 116
  15. 2024Twin-Merging: Dynamic Integration of Modular Expertise in Model Mergingcited by 116
  16. 2021Multi-Center Federated Learning: Clients Clustering for Better Personalizationcited by 113
  17. 2024Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLMcited by 109
  18. 2024CultureBank: An Online Community-Driven Knowledge Base Towards Culturally Aware Language Technologiescited by 98
  19. 2023AdapterSoup: Weight Averaging to Improve Generalization of Pretrained Language Modelscited by 84
  20. 2024Towards Modular LLMs by Building and Reusing a Library of LoRAscited by 72
  21. 2025Whoever Started the Interference Should End It: Guiding Data-Free Model Merging via Task Vectorscited by 62
  22. 2024Dual-Personalizing Adapter for Federated Foundation Modelscited by 55
  23. 2024A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learningcited by 52
  24. 2024LoRA-Flow: Dynamic LoRA Fusion for Large Language Models in Generative Taskscited by 39
  25. 2023Personalized Federated Learning under Mixture of Distributionscited by 37
  26. 2025AdaMMS: Model Merging for Heterogeneous Multimodal Large Language Models with Unsupervised Coefficient Optimizationcited by 25
  27. 2023FedLPA: One-shot Federated Learning with Layer-Wise Posterior Aggregationcited by 22
  28. 2024Closed-form merging of parameter-efficient modules for Federated Continual Learningcited by 22
  29. 2024StyleRemix: Interpretable Authorship Obfuscation via Distillation and Perturbation of Style Elementscited by 18
  30. 2024Retrieval-Augmented Mixture of LoRA Experts for Uploadable Machine Learningcited by 14
  31. 2025MergeME: Model Merging Techniques for Homogeneous and Heterogeneous MoEscited by 13
  32. 2024Task Arithmetic Through The Lens Of One-Shot Federated Learningcited by 12
  33. 2025Personalized Language Models via Privacy-Preserving Evolutionary Model Mergingcited by 8
  34. 2024Local Superior Soups: A Catalyst for Model Merging in Cross-Silo Federated Learningcited by 5
  35. 2022Multi-Model Federated Learningcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025Model Merging via Multi-Teacher Knowledge Distillationcited by 2
  2. 2026Post-Hoc Merging is Not Enough: Many-Shot Model Merging with Loss-Gap Balancingcited by 1
  3. 2026Communication-Efficient Personalized Adaptation via Federated-Local Model Mergingcited by 0
  4. 2026SUM: Unified Geometric Surgery on Spatio-Temporal Adaptation Vectors for Federated Class Incremental Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Global Federated Learning Strategies for Building Efficient Personalized Models
  2. 2022Adaptive Expert Models for Personalization in Federated Learning
  3. 2026FedCoE: Bridging Generalization and Personalization via Federated Coordinated Dual-level MoEs
  4. 2020Specialized federated learning using a mixture of experts
  5. 2026Federated Customization of Large Models: Approaches, Experiments, and Insights
  6. 2024FedD2S: Personalized Data-Free Federated Knowledge Distillation
  7. 2024pFedMoE: Data-Level Personalization with Mixture of Experts for Model-Heterogeneous Personalized Federated Learning
  8. 2023UPFL: Unsupervised Personalized Federated Learning towards New Clients