Data Heterogeneity and Forgotten Labels in Split Federated Learning
Not on the map. Why?
Abstract (arXiv)
In Split Federated Learning (SFL), the clients collaboratively train a model with the help of a server by splitting the model into two parts. Part-1 is trained locally at each client and aggregated by the aggregator at the end of each round. Part-2 is trained at a server that sequentially processes the intermediate activations received from each client. We study the phenomenon of catastrophic forgetting (CF) in SFL in the presence of data heterogeneity. In detail, due to the nature of SFL, local updates of part-1 may drift away from global optima, while part-2 is sensitive to the processing sequence, similar to forgetting in continual learning (CL). Specifically, we observe that the trained model performs better in classes (labels) seen at the end of the sequence. We investigate this phenomenon with emphasis on key aspects of SFL, such as the processing order at the server and the cut layer. Based on our findings, we propose Hydra, a novel mitigation method inspired by multi-head neural networks and adapted for the SFL setting. Extensive numerical evaluations show that Hydra outperforms baselines and methods from the literature.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 37 of its 61 references that are in the map. First posted on arXiv 12 Nov 2025.
Builds on
37 of 61 references.
References · 37
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
- 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
- 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
- 2019Advances and Open Problems in Federated Learningcited by 3,021
- 2018Federated Optimization in Heterogeneous Networkscited by 2,971
- 2018Federated Learning with Non-IID Datacited by 1,099
- 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 897
- 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
- 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 711
- 2021Federated Learning on Non-IID Data Silos: An Experimental Studycited by 566
25 more
- 2018Split learning for health: Distributed deep learning without sharing raw patient datacited by 391
- 2017Measuring Catastrophic Forgetting in Neural Networkscited by 359
- 2021Federated Learning on Non-IID Data: A Surveycited by 345
- 2020SplitFed: When Federated Learning Meets Split Learningcited by 308
- 2020Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semanticscited by 158
- 2021Preservation of the Global Knowledge by Not-True Distillation in Federated Learningcited by 87
- 2022How catastrophic can catastrophic forgetting be in linear regression?cited by 80
- 2023Theory on Forgetting and Generalization of Continual Learningcited by 69
- 2020End-to-End Evaluation of Federated Learning and Split Learning for Internet of Thingscited by 64
- 2023Better Generative Replay for Continual Federated Learningcited by 53
- 2022NISPA: Neuro-Inspired Stability-Plasticity Adaptation for Continual Learning in Sparse Networkscited by 38
- 2024Towards Efficient Replay in Federated Incremental Learningcited by 33
- 2023Continual Learning in Linear Classification on Separable Datacited by 31
- 2022Acceleration of Federated Learning with Alleviated Forgetting in Local Trainingcited by 29
- 2022A Multi-Head Model for Continual Learning via Out-of-Distribution Replaycited by 27
- 2023Convergence Analysis of Sequential Federated Learning on Heterogeneous Datacited by 23
- 2021Is Class-Incremental Enough for Continual Learning?cited by 22
- 2021Evaluation and Optimization of Distributed Machine Learning Techniques for Internet of Thingscited by 20
- 2024Convergence Analysis of Split Federated Learning on Heterogeneous Datacited by 17
- 2020Comparison of Privacy-Preserving Distributed Deep Learning Methods in Healthcarecited by 14
- 2023MergeSFL: Split Federated Learning with Feature Merging and Batch Size Regulationcited by 13
- 2024Continual Learning in the Presence of Repetitioncited by 10
- 2023Split Federated Learning for 6G Enabled-Networks: Requirements, Challenges and Future Directionscited by 8
- 2024Workflow Optimization for Parallel Split Learningcited by 8
- 2021Vulnerability Due to Training Order in Split Learningcited by 2
Built on by
Citing papers per year: 2026: 2.
2 papers.