mlsanityData to 7 Oct 2026

Guided Model Merging for Hybrid Data Learning: Leveraging Centralized Data to Refine Decentralized Models

arXiv 2503.20138 ↗First posted 26 Mar 2025cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Current network training paradigms primarily focus on either centralized or decentralized data regimes. However, in practice, data availability often exhibits a hybrid nature, where both regimes coexist. This hybrid setting presents new opportunities for model training, as the two regimes offer complementary trade-offs: decentralized data is abundant but subject to heterogeneity and communication constraints, while centralized data, though limited in volume and potentially unrepresentative, enables better curation and high-throughput access. Despite its potential, effectively combining these paradigms remains challenging, and few frameworks are tailored to hybrid data regimes. To address this, we propose a novel framework that constructs a model atlas from decentralized models and leverages centralized data to refine a global model within this structured space. The refined model is then used to reinitialize the decentralized models. Our method synergizes federated learning (to exploit decentralized data) and model merging (to utilize centralized data), enabling effective training under hybrid data availability. Theoretically, we show that our approach achieves faster convergence than methods relying solely on decentralized data, due to variance reduction in the merging process. Extensive experiments demonstrate that our framework consistently outperforms purely centralized, purely decentralized, and existing hybrid-adaptable methods. Notably, our method remains robust even when the centralized and decentralized data domains differ or when decentralized data contains noise, significantly broadening its applicability.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 58 of its 72 references that are in the map. First posted on arXiv 26 Mar 2025.

Builds on

58 of 72 references.

References · 58

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  8. 2023Segment Anythingcited by 8,158
  9. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  10. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  11. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  12. 2014Deep Learning Face Attributes in the Wildcited by 3,679
38 more
  1. 2019Advances and Open Problems in Federated Learningcited by 3,021
  2. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  3. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  4. 2022Visual Prompt Tuningcited by 1,674
  5. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  6. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
  7. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
  8. 2022Editing Models with Task Arithmeticcited by 1,076
  9. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,020
  10. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 897
  11. 2019Towards Federated Learning at Scale: System Designcited by 826
  12. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
  13. 2018LEAF: A Benchmark for Federated Settingscited by 780
  14. 2021Model-Contrastive Federated Learningcited by 762
  15. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
  16. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 678
  17. 2020Personalized Federated Learning with Moreau Envelopescited by 612
  18. 2020An Efficient Framework for Clustered Federated Learningcited by 505
  19. 2019FedMD: Heterogenous Federated Learning via Model Distillationcited by 454
  20. 2019Bayesian Nonparametric Federated Learning of Neural Networkscited by 385
  21. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 384
  22. 2019Asynchronous Federated Optimizationcited by 264
  23. 2024Evolutionary Optimization of Model Merging Recipescited by 204
  24. 2021Federated Learning with Buffered Asynchronous Aggregationcited by 188
  25. 2021FedKD: Communication Efficient Federated Learning via Knowledge Distillationcited by 185
  26. 2022FedDC: Federated Learning with Non-IID Data via Local Drift Decoupling and Correctioncited by 181
  27. 2020FedBE: Making Bayesian Model Ensemble Applicable to Federated Learningcited by 176
  28. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Modelscited by 140
  29. 2020Optimal Client Sampling for Federated Learningcited by 123
  30. 2021Parameterized Knowledge Transfer for Personalized Federated Learningcited by 108
  31. 2020TiFL: A Tier-based Federated Learning Systemcited by 95
  32. 2022Personalized Federated Learning via Variational Bayesian Inferencecited by 68
  33. 2020FedAT: A High-Performance and Communication-Efficient Federated Learning System with Asynchronous Tierscited by 64
  34. 2023FedFed: Feature Distillation against Data Heterogeneity in Federated Learningcited by 54
  35. 2024Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costscited by 51
  36. 2023FedASMU: Efficient Asynchronous Federated Learning with Dynamic Staleness-aware Model Updatecited by 32
  37. 2023TimelyFL: Heterogeneity-aware Asynchronous Federated Learning with Adaptive Partial Trainingcited by 27
  38. 2023Confidence-aware Personalized Federated Learning via Variational Expectation Maximizationcited by 17

50 most cited of 58 All 58 (JSON)

Built on by

Not cited yet.