mlsanityData to 14 Oct 2026

On Sampling Strategies for Spectral Model Sharding

arXiv 2410.24106 ↗First posted 31 Oct 2024cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

The problem of heterogeneous clients in federated learning has recently drawn a lot of attention. Spectral model sharding, i.e., partitioning the model parameters into low-rank matrices based on the singular value decomposition, has been one of the proposed solutions for more efficient on-device training in such settings. In this work, we present two sampling strategies for such sharding, obtained as solutions to specific optimization problems. The first produces unbiased estimators of the original weights, while the second aims to minimize the squared approximation error. We discuss how both of these estimators can be incorporated in the federated learning loop and practical considerations that arise during local training. Empirically, we demonstrate that both of these methods can lead to improved performance on various commonly used datasets.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 24 of its 45 references that are in the map. First posted on arXiv 31 Oct 2024.

Builds on

24 of 45 references.

References · 24

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  4. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  5. 2020Adaptive Federated Optimizationcited by 1,046
  6. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 851
  7. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 682
  8. 2019A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protectioncited by 374
  9. 2020HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clientscited by 337
  10. 2019The Non-IID Data Quagmire of Decentralized Machine Learningcited by 305
  11. 2021Escaping the Big Data Paradigm with Compact Transformerscited by 264
  12. 2021FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropoutcited by 192
12 more
  1. 2018Expanding the Reach of Federated Learning by Reducing Client Resource Requirementscited by 163
  2. 2023Heterogeneous Federated Learning: State-of-the-art and Research Challengescited by 161
  3. 2022FedRolex: Model-Heterogeneous Federated Learning with Rolling Sub-Model Extractioncited by 102
  4. 2023SLoRA: Federated Parameter Efficient Fine-Tuning of Language Modelscited by 93
  5. 2022When Federated Learning Meets Pre-trained Language Models' Parameter-Efficient Tuning Methodscited by 75
  6. 2021Initialization and Regularization of Factorized Neural Layerscited by 58
  7. 2022Implicit Regularization Towards Rank Minimization in ReLU Networkscited by 53
  8. 2023Federated Learning for Computationally-Constrained Heterogeneous Devices: A Surveycited by 40
  9. 2021FedHM: Efficient Federated Learning for Heterogeneous Models via Low-rank Factorizationcited by 31
  10. 2022Exploring Low Rank Training of Deep Neural Networkscited by 30
  11. 2023Cuttlefish: Low-Rank Model Training without All the Tuningcited by 25
  12. 2022Federated Learning of Large Models at the Edge via Principal Sub-Model Trainingcited by 13

Built on by

Not cited yet.