mlsanityData to 14 Oct 2026

Grounding Foundation Models through Federated Transfer Learning: A General Framework

arXiv 2311.17431 ↗First posted 29 Nov 2023cs.LG cs.AIcited by 24

Not on the map. Why?

Abstract (arXiv)

Foundation Models (FMs) such as GPT-4 encoded with vast knowledge and powerful emergent abilities have achieved remarkable success in various natural language processing and computer vision tasks. Grounding FMs by adapting them to domain-specific tasks or augmenting them with domain-specific knowledge enables us to exploit the full potential of FMs. However, grounding FMs faces several challenges, stemming primarily from constrained computing resources, data privacy, model heterogeneity, and model ownership. Federated Transfer Learning (FTL), the combination of federated learning and transfer learning, provides promising solutions to address these challenges. In recent years, the need for grounding FMs leveraging FTL, coined FTL-FM, has arisen strongly in both academia and industry. Motivated by the strong growth in FTL-FM research and the potential impact of FTL-FM on industrial applications, we propose an FTL-FM framework that formulates problems of grounding FMs in the federated learning setting, construct a detailed taxonomy based on the FTL-FM framework to categorize state-of-the-art FTL-FM works, and comprehensively overview FTL-FM works based on the proposed taxonomy. We also establish correspondences between FTL-FM and conventional phases of adapting FM so that FM practitioners can align their research works with FTL-FM. In addition, we overview advanced efficiency-improving and privacy-preserving techniques because efficiency and privacy are critical concerns in FTL-FM. Last, we discuss opportunities and future research directions of FTL-FM.

Key facts. Cited by 24 papers in the map (data through 14 Oct 2026). Builds on 107 of its 170 references that are in the map. First posted on arXiv 29 Nov 2023.

Builds on

107 of 170 references.

References · 107

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  7. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  8. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  10. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  11. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  12. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
38 more
  1. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  2. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  3. 2016Deep Learning with Differential Privacycited by 2,728
  4. 2020Extracting Training Data from Large Language Modelscited by 2,189
  5. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  6. 2020Knowledge Distillation: A Surveycited by 1,603
  7. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  8. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  9. 2019Machine Unlearningcited by 1,061
  10. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
  11. 2019Deep Leakage from Gradientscited by 1,021
  12. 2023BloombergGPT: A Large Language Model for Financecited by 939
  13. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
  14. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
  15. 2023Bias and Fairness in Large Language Models: A Surveycited by 639
  16. 2018Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networkscited by 580
  17. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 412
  18. 2021Differentially Private Fine-tuning of Language Modelscited by 400
  19. 2016The CMA Evolution Strategy: A Tutorialcited by 377
  20. 2022Large Language Models Are Reasoning Teacherscited by 329
  21. 2023Provable Robust Watermarking for AI-Generated Textcited by 299
  22. 2022Teaching Small Language Models to Reasoncited by 284
  23. 2020ONION: A Simple and Effective Defense Against Textual Backdoor Attackscited by 263
  24. 2022Black-Box Tuning for Language-Model-as-a-Servicecited by 256
  25. 2023FinGPT: Open-Source Financial Large Language Modelscited by 253
  26. 2022LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learningcited by 230
  27. 2020Information Leakage in Embedding Modelscited by 203
  28. 2023Text Embeddings Reveal (Almost) As Much As Textcited by 192
  29. 2016Practical Secure Aggregation for Federated Learning on User-Held Datacited by 178
  30. 2021Practical and Private (Deep) Learning without Sampling or Shufflingcited by 170
  31. 2020Continual Lifelong Learning in Natural Language Processing: A Surveycited by 152
  32. 2023FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learningcited by 149
  33. 2023Who Wrote this Code? Watermarking for Code Generationcited by 141
  34. 2022Explanations from Large Language Models Make Small Reasoners Bettercited by 124
  35. 2023LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructionscited by 124
  36. 2023Is ChatGPT Fair for Recommendation? Evaluating Fairness in Large Language Model Recommendationcited by 114
  37. 2020BERT-of-Theseus: Compressing BERT by Progressive Module Replacingcited by 113
  38. 2022Synthetic Text Generation with Differential Privacy: A Simple and Practical Recipecited by 108

50 most cited of 107 All 107 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 13; 2025: 9; 2026: 2.

24 papers.

Sort:

By area:

  1. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 86
  2. 2024Advances and Open Challenges in Federated Foundation Modelscited by 36
  3. 2024Federated Large Language Models: Current Progress and Future Directionscited by 27
  4. 2024Personalized Federated Continual Learning via Multi-granularity Promptcited by 20
  5. 2025Ten Challenging Problems in Federated Foundation Modelscited by 20
  6. 2024FedMKT: Federated Mutual Knowledge Transfer for Large and Small Language Modelscited by 19
  7. 2024Synergizing Foundation Models and Federated Learning: A Surveycited by 14
  8. 2024No Free Lunch Theorem for Privacy-Preserving LLM Inferencecited by 13
  9. 2025Distributed LLMs and Multimodal Large Language Models: A Survey on Advances, Challenges, and Future Directionscited by 10
  10. 2024Handling Spatial-Temporal Data Heterogeneity for Federated Continual Learning via Tail Anchorcited by 9
  11. 2025AugFL: Augmenting Federated Learning with Pretrained Modelscited by 6
  12. 2024Federated Co-tuning Framework for Large and Small Language Modelscited by 5
12 more
  1. 2024A Taxonomy of Architecture Options for Foundation Model-based Agents: Analysis and Decision Modelcited by 4
  2. 2024FedEval-LLM: Federated Evaluation of Large Language Models on Downstream Tasks with Collective Wisdomcited by 3
  3. 2024Review of Mathematical Optimization in Federated Learningcited by 3
  4. 2025A Survey on Foundation Models for Personalized Federated Intelligencecited by 3
  5. 2024Federated Domain-Specific Knowledge Transfer on Large Language Models Using Synthetic Datacited by 2
  6. 2025Efficient Federated Learning with Heterogeneous Data and Adaptive Dropoutcited by 2
  7. 2025Foundational Models and Federated Learning: Survey, Taxonomy, Challenges and Practical Insightscited by 1
  8. 2025A Survey of Personalized Federated Foundation Models for Privacy-Preserving Recommendationcited by 0
  9. 2025Large-Small Model Collaborative Framework for Federated Continual Learningcited by 0
  10. 2025A Scenario-Oriented Survey of Federated Recommender Systems: Techniques, Challenges, and Future Directionscited by 0
  11. 2026FedGRPO: Privately Optimizing Foundation Models with Group-Relative Rewards from Domain Clientcited by 0
  12. 2026FedProxy: Federated Fine-Tuning of LLMs via Proxy SLMs and Heterogeneity-Aware Fusioncited by 0