mlsanityData to 7 Oct 2026

Local Superior Soups: A Catalyst for Model Merging in Cross-Silo Federated Learning

arXiv 2410.23660 ↗First posted 31 Oct 2024cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

Federated learning (FL) is a learning paradigm that enables collaborative training of models using decentralized data. Recently, the utilization of pre-trained weight initialization in FL has been demonstrated to effectively improve model performance. However, the evolving complexity of current pre-trained models, characterized by a substantial increase in parameters, markedly intensifies the challenges associated with communication rounds required for their adaptation to FL. To address these communication cost issues and increase the performance of pre-trained model adaptation in FL, we propose an innovative model interpolation-based local training technique called ``Local Superior Soups.'' Our method enhances local training across different clients, encouraging the exploration of a connected low-loss basin within a few communication rounds through regularized model interpolation. This approach acts as a catalyst for the seamless adaptation of pre-trained models in in FL. We demonstrated its effectiveness and efficiency across diverse widely-used FL datasets. Our code is available at https://github.com/ubc-tea/Local-Superior-Soups.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 48 of its 58 references that are in the map. First posted on arXiv 31 Oct 2024.

Builds on

48 of 58 references.

References · 48

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  9. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  10. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  11. 2014Unsupervised Domain Adaptation by Backpropagationcited by 1,906
  12. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
36 more
  1. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,327
  2. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  3. 2019Deep Leakage from Gradientscited by 1,004
  4. 2021Model-Contrastive Federated Learningcited by 762
  5. 2018On Lazy Training in Differentiable Programmingcited by 671
  6. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  7. 2021FedBN: Federated Learning on Non-IID Features via Local Batch Normalizationcited by 565
  8. 2019A Simple Baseline for Bayesian Uncertainty in Deep Learningcited by 555
  9. 2021Exploiting Shared Representations for Personalized Federated Learningcited by 537
  10. 2021Towards Personalized Federated Learningcited by 462
  11. 2021SWAD: Domain Generalization by Seeking Flat Minimacited by 431
  12. 2018Rethinking ImageNet Pre-trainingcited by 376
  13. 2021A Field Guide to Federated Optimizationcited by 312
  14. 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 291
  15. 2019Uniform convergence may be unable to explain generalization in deep learningcited by 223
  16. 2020Personalized Federated Learning with First Order Model Optimizationcited by 201
  17. 2021FedBABU: Towards Enhanced Representation for Federated Image Classificationcited by 180
  18. 2024Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuningcited by 174
  19. 2022Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learningcited by 172
  20. 2022Diverse Weight Averaging for Out-of-Distribution Generalizationcited by 148
  21. 2019One-Shot Federated Learningcited by 124
  22. 2021DENSE: Data-Free One-Shot Federated Learningcited by 100
  23. 2019Subspace Inference for Bayesian Deep Learningcited by 94
  24. 2023Deep Model Fusion: A Surveycited by 89
  25. 2021Learning Neural Network Subspacescited by 82
  26. 2022When Federated Learning Meets Pre-trained Language Models' Parameter-Efficient Tuning Methodscited by 74
  27. 2022Where to Begin? On the Impact of Pre-Training and Initialization in Federated Learningcited by 71
  28. 2023FedSpeed: Larger Local Interval, Less Communication Round, and Higher Generalization Accuracycited by 49
  29. 2020Affinity and Diversity: Quantifying Mechanisms of Data Augmentationcited by 45
  30. 2024FedLLM-Bench: Realistic Benchmarks for Federated Learning of Large Language Modelscited by 26
  31. 2021Cost-Effective Federated Learning in Mobile Edge Networkscited by 25
  32. 2022Where to start? Analyzing the potential value of intermediate modelscited by 23
  33. 2023FedSoup: Improving Generalization and Personalization in Federated Learning via Selective Model Interpolationcited by 14
  34. 2022VITA: A Multi-Source Vicinal Transfer Augmentation Method for Out-of-Distribution Generalizationcited by 3
  35. 2023Network Adaptive Federated Learning: Congestion and Lossy Compressioncited by 3
  36. 2021Experiments with Rich Regime Training for Deep Learningcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 1.

5 papers.

  1. 2025Modeling Multi-Task Model Merging as Adaptive Projective Gradient Descentcited by 39
  2. 2025Can Textual Gradient Work in Federated Learning?cited by 14
  3. 2025FedMerge: Federated Personalization via Model Mergingcited by 4
  4. 2026Self-Soupervision: Cooking Model Soups without Labelscited by 1
  5. 2025A Closer Look at Personalized Fine-Tuning in Heterogeneous Federated Learningcited by 0