mlsanityData to 7 Oct 2026

FedTLU: Federated Learning with Targeted Layer Updates

arXiv 2412.17692 ↗First posted 23 Dec 2024cs.LG cs.AI cs.DCcited by 0

Not on the map. Why?

Abstract (arXiv)

Federated learning (FL) addresses privacy concerns in training language models by enabling multiple clients to contribute to the training, without sending their data to others. However, non-IID (identically and independently distributed) data across clients often limits FL's performance. This issue is especially challenging during model fine-tuning, as noise due to variations in clients' data distributions can harm model convergence near stationary points. This paper proposes a targeted layer update strategy for fine-tuning in FL. Instead of randomly updating layers of the language model, as often done in practice, we use a scoring mechanism to identify and update the most critical layers, avoiding excessively noisy or even poisoned updates by freezing the parameters in other layers. We show in extensive experiments that our method improves convergence and performance in non-IID settings, offering a more efficient approach to fine-tuning federated language models.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 29 of its 45 references that are in the map. First posted on arXiv 23 Dec 2024.

Builds on

29 of 45 references.

References · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  4. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  5. 2019Advances and Open Problems in Federated Learningcited by 3,021
  6. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  7. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
  8. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  9. 2017Federated Multi-Task Learningcited by 669
  10. 2020Ditto: Fair and Robust Federated Learning Through Personalizationcited by 665
  11. 2019Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization under Privacy Constraintscited by 499
  12. 2021Towards Personalized Federated Learningcited by 462
17 more
  1. 2021Differentially Private Fine-tuning of Language Modelscited by 392
  2. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 380
  3. 2019Transfusion: Understanding Transfer Learning for Medical Imagingcited by 366
  4. 2021Federated Learning on Non-IID Data: A Surveycited by 345
  5. 2019Momentum-Based Variance Reduction in Non-Convex SGDcited by 302
  6. 2018Sensitivity and Generalization in Neural Networks: an Empirical Studycited by 187
  7. 2023Heterogeneous Federated Learning: State-of-the-art and Research Challengescited by 160
  8. 2022Generalized Federated Learning via Sharpness Aware Minimizationcited by 144
  9. 2021Local Learning Matters: Rethinking Data Heterogeneity in Federated Learningcited by 114
  10. 2018On the importance of single directions for generalizationcited by 99
  11. 2023Sensitivity-Aware Visual Parameter-Efficient Fine-Tuningcited by 74
  12. 2024FedBiOT: LLM Local Fine-tuning in Federated Learning without Full Modelcited by 73
  13. 2023ZiCo: Zero-shot NAS via Inverse Coefficient of Variation on Gradientscited by 57
  14. 2021Scaling Federated Learning for Fine-tuning of Large Language Modelscited by 36
  15. 2022Personalization Improves Privacy-Accuracy Tradeoffs in Federated Learningcited by 23
  16. 2016Improved Dropout for Shallow and Deep Learningcited by 11
  17. 2024FedBaF: Federated Learning Aggregation Biased by a Foundation Modelcited by 3

Built on by

Not cited yet.