# Efficient Split-Mix Federated Learning for On-Demand and In-Situ Customization

arXiv: https://arxiv.org/abs/2203.09747 · HTML: https://mlsanity.com/paper/2203.09747/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 49 references that are in the map.
- First posted on arXiv 18 Mar 2022.

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Unsupervised Domain Adaptation by Backpropagation](https://mlsanity.com/paper/1409.7495/) (2014)
- [Moment Matching for Multi-Source Domain Adaptation](https://mlsanity.com/paper/1812.01754/) (2018)
- [Robustness May Be at Odds with Accuracy](https://mlsanity.com/paper/1805.12152/) (2018)
- [LEAF: A Benchmark for Federated Settings](https://mlsanity.com/paper/1812.01097/) (2018)
- [Ensemble Distillation for Robust Model Fusion in Federated Learning](https://mlsanity.com/paper/2006.07242/) (2020)
- [FedBN: Federated Learning on Non-IID Features via Local Batch Normalization](https://mlsanity.com/paper/2102.07623/) (2021)
- [Data-Free Knowledge Distillation for Heterogeneous Federated Learning](https://mlsanity.com/paper/2105.10056/) (2021)
- [HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clients](https://mlsanity.com/paper/2010.01264/) (2020)
- [SkipNet: Learning Dynamic Routing in Convolutional Networks](https://mlsanity.com/paper/1711.09485/) (2017)
- [Personalized Federated Learning: A Meta-Learning Approach](https://mlsanity.com/paper/2002.07948/) (2020)
- [Slimmable Neural Networks](https://mlsanity.com/paper/1812.08928/) (2018)
- [Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning](https://mlsanity.com/paper/2012.09816/) (2020)
- [Federated Optimization:Distributed Optimization Beyond the Datacenter](https://mlsanity.com/paper/1511.03575/) (2015)
- [BlockDrop: Dynamic Inference Paths in Residual Networks](https://mlsanity.com/paper/1711.08393/) (2017)
- [Federated Adversarial Domain Adaptation](https://mlsanity.com/paper/1911.02054/) (2019)
- [AugMax: Adversarial Composition of Random Augmentations for Robust Training](https://mlsanity.com/paper/2110.13771/) (2021)
- [AI Benchmark: Running Deep Neural Networks on Android Smartphones](https://mlsanity.com/paper/1810.01109/) (2018)
- [Dynamic Deep Neural Networks: Optimizing Accuracy-Efficiency Trade-offs by Selective Execution](https://mlsanity.com/paper/1701.00299/) (2017)
- [Once-for-All Adversarial Training: In-Situ Tradeoff between Robustness and Accuracy for Free](https://mlsanity.com/paper/2010.11828/) (2020)
- [A First Look at Deep Learning Apps on Smartphones](https://mlsanity.com/paper/1812.05448/) (2018)
- [Fed-ensemble: Improving Generalization through Model Ensembling in Federated Learning](https://mlsanity.com/paper/2107.10663/) (2021)

## Built on by (most cited first, 30 of 35)

- [Heterogeneous Federated Learning: State-of-the-art and Research Challenges](https://mlsanity.com/paper/2307.10616/) (2023)
- [DisPFL: Towards Communication-Efficient Personalized Federated Learning via Decentralized Sparse Training](https://mlsanity.com/paper/2206.00187/) (2022)
- [A Survey on Heterogeneous Federated Learning](https://mlsanity.com/paper/2210.04505/) (2022)
- [Efficient Personalized Federated Learning via Sparse Model-Adaptation](https://mlsanity.com/paper/2305.02776/) (2023)
- [Personalized Federated Learning under Mixture of Distributions](https://mlsanity.com/paper/2305.01068/) (2023)
- [DFRD: Data-Free Robustness Distillation for Heterogeneous Federated Learning](https://mlsanity.com/paper/2309.13546/) (2023)
- [A Survey of What to Share in Federated Learning: Perspectives on Model Utility, Privacy Leakage, and Communication Efficiency](https://mlsanity.com/paper/2307.10655/) (2023)
- [Federated Learning of Large Models at the Edge via Principal Sub-Model Training](https://mlsanity.com/paper/2208.13141/) (2022)
- [Recurrent Early Exits for Federated Learning with Heterogeneous Clients](https://mlsanity.com/paper/2405.14791/) (2024)
- [GNN at the Edge: Cost-Efficient Graph Neural Network Processing over Distributed Edge Servers](https://mlsanity.com/paper/2210.17281/) (2022)
- [Who Leaked the Model? Tracking IP Infringers in Accountable Federated Learning](https://mlsanity.com/paper/2312.03205/) (2023)
- [NeFL: Nested Model Scaling for Federated Learning with System Heterogeneous Clients](https://mlsanity.com/paper/2308.07761/) (2023)
- [Personalised Federated Learning On Heterogeneous Feature Spaces](https://mlsanity.com/paper/2301.11447/) (2023)
- [AdapterFL: Adaptive Heterogeneous Federated Learning for Resource-constrained Mobile Computing Systems](https://mlsanity.com/paper/2311.14037/) (2023)
- [FedSAC: Dynamic Submodel Allocation for Collaborative Fairness in Federated Learning](https://mlsanity.com/paper/2405.18291/) (2024)
- [Exploring the Practicality of Federated Learning: A Survey Towards the Communication Perspective](https://mlsanity.com/paper/2405.20431/) (2024)
- [Outsourcing Training without Uploading Data via Efficient Collaborative Open-Source Sampling](https://mlsanity.com/paper/2210.12575/) (2022)
- [HierarchyFL: Heterogeneous Federated Learning via Hierarchical Self-Distillation](https://mlsanity.com/paper/2212.02006/) (2022)
- [CC-FedAvg: Computationally Customized Federated Averaging](https://mlsanity.com/paper/2212.13679/) (2022)
- [DFDG: Data-Free Dual-Generator Adversarial Distillation for One-Shot Federated Learning](https://mlsanity.com/paper/2409.07734/) (2024)
- [SFL-LEO: Asynchronous Split-Federated Learning Design for LEO Satellite-Ground Network Framework](https://mlsanity.com/paper/2504.13479/) (2025)
- [Toward Enhancing Representation Learning in Federated Multi-Task Settings](https://mlsanity.com/paper/2602.01626/) (2026)
- [Reducing Training Time in Cross-Silo Federated Learning using Multigraph Topology](https://mlsanity.com/paper/2207.09657/) (2022)
- [Breaking On-device Training Memory Wall: A Systematic Survey](https://mlsanity.com/paper/2306.10388/) (2023)
- [ECLM: Efficient Edge-Cloud Collaborative Learning with Continuous Environment Adaptation](https://mlsanity.com/paper/2311.11083/) (2023)
- [FedTrans: Efficient Federated Learning via Multi-Model Transformation](https://mlsanity.com/paper/2404.13515/) (2024)
- [Heterogeneity-Aware Coordination for Federated Learning via Stitching Pre-trained blocks](https://mlsanity.com/paper/2409.07202/) (2024)
- [FedMHO: Heterogeneous One-Shot Federated Learning Towards Resource-Constrained Edge Devices](https://mlsanity.com/paper/2502.08518/) (2025)
- [ASFL: An Adaptive Model Splitting and Resource Allocation Framework for Split Federated Learning](https://mlsanity.com/paper/2603.04437/) (2026)
- [When Computing Power Network Meets Distributed Machine Learning: An Efficient Federated Split Learning Framework](https://mlsanity.com/paper/2305.12979/) (2023)

## Abstract (arXiv)

Federated learning (FL) provides a distributed learning framework for multiple participants to collaborate learning without sharing raw data. In many practical FL scenarios, participants have heterogeneous resources due to disparities in hardware and inference dynamics that require quickly loading models of different sizes and levels of robustness. The heterogeneity and dynamics together impose significant challenges to existing FL approaches and thus greatly limit FL's applicability. In this paper, we propose a novel Split-Mix FL strategy for heterogeneous participants that, once training is done, provides in-situ customization of model sizes and robustness. Specifically, we achieve customization by learning a set of base sub-networks of different sizes and robustness levels, which are later aggregated on-demand according to inference requirements. This split-mix strategy achieves customization with high efficiency in communication, storage, and inference. Extensive experiments demonstrate that our method provides better in-situ customization than the existing heterogeneous-architecture FL methods. Codes and pre-trained models are available: https://github.com/illidanlab/SplitMix.