Addressing the Collaboration Dilemma in Low-Data Federated Learning via Transient Sparsity
Not on the map. Why?
Abstract (arXiv)
Federated learning (FL) enables collaborative model training across decentralized clients while preserving data privacy, leveraging aggregated updates to build robust global models. However, this training paradigm faces significant challenges due to data heterogeneity and limited local datasets, which often impede effective collaboration. In such scenarios, we identify the Layer-wise Inertia Phenomenon in FL, wherein the middle layers of global model undergo minimal updates after early communication rounds, ultimately limiting the effectiveness of global aggregation. We demonstrate the presence of this phenomenon across a wide range of federated settings, spanning diverse datasets and architectures. To address this issue, we propose LIPS (Layer-wise Inertia Phenomenon with Sparsity), a simple yet effective method that periodically introduces transient sparsity to stimulate meaningful updates and empower global aggregation. Experiments demonstrate that LIPS effectively mitigates layer-wise inertia, enhances aggregation effectiveness, and improves overall performance in various FL scenarios. This work not only deepens the understanding of layer-wise learning dynamics in FL but also paves the way for more effective collaboration strategies in resource-constrained environments. Our code is publicly available at: https://github.com/QiaoXiao7282/LIPS.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 59 of its 68 references that are in the map. First posted on arXiv 1 Jun 2025.
Builds on
59 of 68 references.
References · 59
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
- 2018Federated Optimization in Heterogeneous Networkscited by 2,987
- 2018Federated Learning with Non-IID Datacited by 1,108
- 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 851
- 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 812
- 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 682
- 2017Federated Multi-Task Learningcited by 672
- 2020Ditto: Fair and Robust Federated Learning Through Personalizationcited by 672
- 2020The Future of Digital Health with Federated Learningcited by 633
- 2021FedBN: Federated Learning on Non-IID Features via Local Batch Normalizationcited by 571
38 more
- 2021Federated Learning on Non-IID Data Silos: An Experimental Studycited by 568
- 2019Federated Learning with Personalization Layerscited by 541
- 2021Exploiting Shared Representations for Personalized Federated Learningcited by 541
- 2020An Efficient Framework for Clustered Federated Learningcited by 511
- 2019Rigging the Lottery: Making All Tickets Winnerscited by 495
- 2021Towards Personalized Federated Learningcited by 464
- 2017Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Sciencecited by 328
- 2020Three Approaches for Personalization with Applications to Federated Learningcited by 325
- 2019Improving Federated Learning Personalization via Model Agnostic Meta Learningcited by 269
- 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 226
- 2019Model Pruning Enables Efficient Federated Learning on Edge Devicescited by 222
- 2017Gradient Sparsification for Communication-Efficient Distributed Optimizationcited by 201
- 2021Federated Learning for Smart Healthcare: A Surveycited by 184
- 2020Salvaging Federated Learning by Local Adaptationcited by 142
- 2022More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsitycited by 142
- 2021Federated Learning for Open Bankingcited by 141
- 2021Linear Convergence in Federated Learning: Tackling Client Heterogeneity and Sparse Gradientscited by 123
- 2022FedALA: Adaptive Local Aggregation for Personalized Federated Learningcited by 122
- 2021Parameterized Knowledge Transfer for Personalized Federated Learningcited by 109
- 2021FedMix: Approximation of Mixup under Mean Augmented Federated Learningcited by 108
- 2022Layer-wised Model Aggregation for Personalized Federated Learningcited by 96
- 2021Preservation of the Global Knowledge by Not-True Distillation in Federated Learningcited by 87
- 2020Adaptive Gradient Sparsification for Efficient Federated Learning: An Online Learning Approachcited by 81
- 2021Federated Reconstruction: Partially Local Federated Learningcited by 79
- 2020Gradient Flow in Sparse Neural Networks and How Lottery Tickets Wincited by 77
- 2022DisPFL: Towards Communication-Efficient Personalized Federated Learning via Decentralized Sparse Trainingcited by 74
- 2021What Do We Mean by Generalization in Federated Learning?cited by 62
- 2021Federated Dropout -- A Simple Approach for Enabling Federated Learning on Resource Constrained Devicescited by 56
- 2021Federated Dynamic Sparse Training: Computing Less, Communicating Less, Yet Learning Bettercited by 55
- 2021Flexible Clustered Federated Learning for Client-Level Data Distribution Shiftcited by 47
- 2022Sparsity Winning Twice: Better Robust Generalization from More Efficient Trainingcited by 41
- 2023Efficient Personalized Federated Learning via Sparse Model-Adaptationcited by 38
- 2024Tracing Representation Progression: Analyzing and Enhancing Layer-Wise Similaritycited by 36
- 2024Streamlining Redundant Layers to Compress Large Language Modelscited by 35
- 2022Achieving Personalized Federated Learning with Sparse Local Modelscited by 31
- 2024FedSelect: Personalized Federated Learning with Customized Selection of Parameters for Fine-Tuningcited by 27
- 2024Federated LoRA with Sparse Communicationcited by 26
- 2020Federated Residual Learningcited by 23
50 most cited of 59 All 59 (JSON)
Built on by
Not cited yet.