mlsanityData to 7 Oct 2026

AugFL: Augmenting Federated Learning with Pretrained Models

arXiv 2503.02154 ↗First posted 4 Mar 2025cs.LG cs.AI cs.DCcited by 6

Not on the map. Why?

Abstract (arXiv)

Federated Learning (FL) has garnered widespread interest in recent years. However, owing to strict privacy policies or limited storage capacities of training participants such as IoT devices, its effective deployment is often impeded by the scarcity of training data in practical decentralized learning environments. In this paper, we study enhancing FL with the aid of (large) pre-trained models (PMs), that encapsulate wealthy general/domain-agnostic knowledge, to alleviate the data requirement in conducting FL from scratch. Specifically, we consider a networked FL system formed by a central server and distributed clients. First, we formulate the PM-aided personalized FL as a regularization-based federated meta-learning problem, where clients join forces to learn a meta-model with knowledge transferred from a private PM stored at the server. Then, we develop an inexact-ADMM-based algorithm, AugFL, to optimize the problem with no need to expose the PM or incur additional computational costs to local clients. Further, we establish theoretical guarantees for AugFL in terms of communication complexity, adaptation performance, and the benefit of knowledge transfer in general non-convex cases. Extensive experiments corroborate the efficacy and superiority of AugFL over existing baselines.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 64 of its 84 references that are in the map. First posted on arXiv 4 Mar 2025.

Builds on

64 of 84 references.

References · 64

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  7. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  8. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  9. 2019Advances and Open Problems in Federated Learningcited by 3,021
  10. 2022Emergent Abilities of Large Language Modelscited by 2,483
  11. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,531
  12. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,461
38 more
  1. 2019A Comprehensive Survey on Transfer Learningcited by 1,154
  2. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
  3. 2019Relational Knowledge Distillationcited by 873
  4. 2019Towards Federated Learning at Scale: System Designcited by 826
  5. 2019Contrastive Representation Distillationcited by 734
  6. 2017Federated Multi-Task Learningcited by 669
  7. 2019Federated Learning with Personalization Layerscited by 533
  8. 2021Towards Personalized Federated Learningcited by 462
  9. 2021Differentially Private Fine-tuning of Language Modelscited by 392
  10. 2022Decoupled Knowledge Distillationcited by 380
  11. 2017A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasetscited by 374
  12. 2021FedProto: Federated Prototype Learning across Heterogeneous Clientscited by 349
  13. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  14. 2019Improving Federated Learning Personalization via Model Agnostic Meta Learningcited by 264
  15. 2021Federated Learning for Internet of Things: A Comprehensive Surveycited by 247
  16. 2018Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Datacited by 242
  17. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 193
  18. 2018Federated Meta-Learning with Fast Convergence and Efficient Communicationcited by 154
  19. 2020FedPD: A Federated Learning Framework with Optimal Rates and Adaptivity to Non-IID Datacited by 143
  20. 2022Federated Learning from Pre-Trained Models: A Contrastive Learning Approachcited by 141
  21. 2019On the Convergence Theory of Gradient-Based Model-Agnostic Meta-Learning Algorithmscited by 132
  22. 2015Global Convergence of ADMM in Nonconvex Nonsmooth Optimizationcited by 98
  23. 2019Federated Learning over Wireless Networks: Convergence Analysis and Resource Allocationcited by 93
  24. 2023Offsite-Tuning: Transfer Learning without Full Modelcited by 77
  25. 2021Show, Attend and Distill:Knowledge Distillation via Attention-based Feature Matchingcited by 76
  26. 2022Where to Begin? On the Impact of Pre-Training and Initialization in Federated Learningcited by 71
  27. 2014Convergence Analysis of Alternating Direction Method of Multipliers for a Family of Nonconvex Problemscited by 64
  28. 2023FATE-LLM: A Industrial Grade Federated Learning Framework for Large Language Modelscited by 62
  29. 2021Federated Learning for Internet of Things: Applications, Challenges, and Opportunitiescited by 49
  30. 2019Mitigating Information Leakage in Image Representations: A Maximum Entropy Approachcited by 48
  31. 2023FedCP: Separating Feature Information for Personalized Federated Learning via Conditional Policycited by 36
  32. 2021FedIPR: Ownership Verification for Federated Deep Neural Network Modelscited by 33
  33. 2022Just Fine-tune Twice: Selective Differential Privacy for Large Language Modelscited by 32
  34. 2023Can Public Large Language Models Help Private Cross-device Federated Learning?cited by 30
  35. 2023Mobile Foundation Model as Firmwarecited by 25
  36. 2023Grounding Foundation Models through Federated Transfer Learning: A General Frameworkcited by 24
  37. 2020Inexact-ADMM Based Federated Meta-Learning for Fast and Continual Edge Learningcited by 22
  38. 2023Selective Pre-training for Private Fine-tuningcited by 22

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 2.

6 papers.

  1. 2025Efficient Federated Learning with Heterogeneous Data and Adaptive Dropoutcited by 2
  2. 2025Graph Representation-based Model Poisoning on the Heterogeneous Internet of Agentscited by 2
  3. 2025Task-agnostic Low-rank Residual Adaptation for Efficient Federated Continual Fine-Tuningcited by 1
  4. 2025FedHiP: Heterogeneity-Invariant Personalized Federated Learning Through Closed-Form Solutionscited by 0
  5. 2026QuantFL: Sustainable Federated Learning for Edge IoT via Pre-Trained Model Quantisationcited by 0
  6. 2026Graph Representation Learning Augmented Model Manipulation on Federated Fine-Tuning of LLMscited by 0