Shift Happens: Mixture of Experts based Continual Adaptation in Federated Learning

arXiv 2506.18789 ↗First posted 23 Jun 2025cs.LG cs.AIcited by 0

On the map: Mixture-of-experts → Federated mixture-of-experts

Abstract (arXiv)

Federated Learning (FL) enables collaborative model training across decentralized clients without sharing raw data, yet faces significant challenges in real-world settings where client data distributions evolve dynamically over time. This paper tackles the critical problem of covariate and label shifts in streaming FL environments, where non-stationary data distributions degrade model performance and necessitate a middleware layer that adapts FL to distributional shifts. We introduce ShiftEx, a shift-aware mixture of experts framework that dynamically creates and trains specialized global models in response to detected distribution shifts using Maximum Mean Discrepancy for covariate shifts. The framework employs a latent memory mechanism for expert reuse and implements facility location-based optimization to jointly minimize covariate mismatch, expert creation costs, and label imbalance. Through theoretical analysis and comprehensive experiments on benchmark datasets, we demonstrate 5.5-12.9 percentage point accuracy improvements and 22-95 % faster adaptation compared to state-of-the-art FL baselines across diverse shift scenarios. The proposed approach offers a scalable, privacy-preserving middleware solution for FL systems operating in non-stationary, real-world conditions while minimizing communication and computational overhead.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Federated mixture-of-experts, one of 12 directions in Mixture-of-experts. Builds on 46 of its 94 references that are in the map. First posted on arXiv 23 Jun 2025.

Builds on

46 of 94 references.

Closest ancestors · 5

Same direction, newest first

2024FedMoE: Personalized Federated Learning via Heterogeneous Mixture of Experts21 Aug
2021Federated Mixture of Experts14 Jul
2021Robust Federated Learning by Mixture of Experts23 Apr
2020PFL-MoE: Personalized Federated Learning Based on Mixture of Experts31 Dec
2020Specialized federated learning using a mixture of experts5 Oct

Other references · 41

2015Deep Residual Learning for Image Recognitioncited by 52,096
2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
2016Densely Connected Convolutional Networkscited by 6,720
2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
2019Advances and Open Problems in Federated Learningcited by 3,046
2018Federated Optimization in Heterogeneous Networkscited by 2,987
2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
2016Deep Learning with Differential Privacycited by 2,728
2018Federated Learning with Non-IID Datacited by 1,108
2019Deep Leakage from Gradientscited by 1,021
2018LEAF: A Benchmark for Federated Settingscited by 792
2018How To Backdoor Federated Learningcited by 765

29 more

2020Inverting Gradients -- How easy is it to break privacy in federated learning?cited by 712
2021FedBN: Federated Learning on Non-IID Features via Local Batch Normalizationcited by 571
2018Exploiting Unintended Feature Leakage in Collaborative Learningcited by 520
2020An Efficient Framework for Clustered Federated Learningcited by 511
2020Learning under Concept Drift: A Reviewcited by 477
2020Flower: A Friendly Federated Learning Research Frameworkcited by 455
2020iDLG: Improved Deep Leakage from Gradientscited by 382
2017Functional Map of the Worldcited by 338
2021See through Gradients: Image Batch Recovery via GradInversioncited by 326
2021Membership Inference Attacks on Machine Learning: A Surveycited by 321
2020Federated Continual Learning with Weighted Inter-client Transfercited by 161
2021FedMix: Approximation of Mixup under Mean Augmented Federated Learningcited by 108
2023Federated Continual Learning via Knowledge Fusion: A Surveycited by 53
2021Flexible Clustered Federated Learning for Client-Level Data Distribution Shiftcited by 47
2023Personalized Federated Learning under Mixture of Distributionscited by 37
2021Non-IID data and Continual Learning processes in Federated Learning: A long road aheadcited by 36
2022Federated Learning under Distributed Concept Driftcited by 36
2021Concept drift detection and adaptation for federated and continual learningcited by 32
2023Adaptive Test-Time Personalization for Federated Learningcited by 23
2022Rethinking Normalization Methods in Federated Learningcited by 19
2023Federated Learning for Data Streamscited by 16
2025Federated Continual Learning: Concepts, Challenges, and Solutionscited by 13
2024Federated Continual Learning for Edge-AI: A Comprehensive Surveycited by 11
2023On the Local Cache Update Rules in Streaming Federated Learningcited by 8
2023Federated Learning under Covariate Shifts with Generalization Guaranteescited by 6
2020Evaluation Framework For Large-scale Federated Learningcited by 4
2024Federated Learning Clients Clustering with Adaptation to Data Driftscited by 3
2023FLIPS: Federated Learning using Intelligent Participant Selectioncited by 2
2023Federated Covariate Shift Adaptation for Missing Target Output Valuescited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way