# Grounding Foundation Models through Federated Transfer Learning: A General Framework

arXiv: https://arxiv.org/abs/2311.17431 · HTML: https://mlsanity.com/paper/2311.17431/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 107 of its 170 references that are in the map.
- First posted on arXiv 29 Nov 2023.

## Builds on (most cited first, 30 of 107)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models](https://mlsanity.com/paper/2106.10199/) (2021)
- [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022)
- [Machine Unlearning](https://mlsanity.com/paper/1912.03817/) (2019)
- [GLM: General Language Model Pretraining with Autoregressive Blank Infilling](https://mlsanity.com/paper/2103.10360/) (2021)
- [Deep Leakage from Gradients](https://mlsanity.com/paper/1906.08935/) (2019)
- [BloombergGPT: A Large Language Model for Finance](https://mlsanity.com/paper/2303.17564/) (2023)
- [Towards a Unified View of Parameter-Efficient Transfer Learning](https://mlsanity.com/paper/2110.04366/) (2021)
- [Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes](https://mlsanity.com/paper/2305.02301/) (2023)
- [Bias and Fairness in Large Language Models: A Survey](https://mlsanity.com/paper/2309.00770/) (2023)
- [Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networks](https://mlsanity.com/paper/1805.12185/) (2018)
- [Large Language Models Can Be Strong Differentially Private Learners](https://mlsanity.com/paper/2110.05679/) (2021)
- [Differentially Private Fine-tuning of Language Models](https://mlsanity.com/paper/2110.06500/) (2021)

## Built on by (most cited first, 24 of 24)

- [A Survey of Resource-efficient LLM and Multimodal Foundation Models](https://mlsanity.com/paper/2401.08092/) (2024)
- [Advances and Open Challenges in Federated Foundation Models](https://mlsanity.com/paper/2404.15381/) (2024)
- [Federated Large Language Models: Current Progress and Future Directions](https://mlsanity.com/paper/2409.15723/) (2024)
- [Personalized Federated Continual Learning via Multi-granularity Prompt](https://mlsanity.com/paper/2407.00113/) (2024)
- [Ten Challenging Problems in Federated Foundation Models](https://mlsanity.com/paper/2502.12176/) (2025)
- [FedMKT: Federated Mutual Knowledge Transfer for Large and Small Language Models](https://mlsanity.com/paper/2406.02224/) (2024)
- [Synergizing Foundation Models and Federated Learning: A Survey](https://mlsanity.com/paper/2406.12844/) (2024)
- [No Free Lunch Theorem for Privacy-Preserving LLM Inference](https://mlsanity.com/paper/2405.20681/) (2024)
- [Distributed LLMs and Multimodal Large Language Models: A Survey on Advances, Challenges, and Future Directions](https://mlsanity.com/paper/2503.16585/) (2025)
- [Handling Spatial-Temporal Data Heterogeneity for Federated Continual Learning via Tail Anchor](https://mlsanity.com/paper/2412.18355/) (2024)
- [AugFL: Augmenting Federated Learning with Pretrained Models](https://mlsanity.com/paper/2503.02154/) (2025)
- [Federated Co-tuning Framework for Large and Small Language Models](https://mlsanity.com/paper/2411.11707/) (2024)
- [A Taxonomy of Architecture Options for Foundation Model-based Agents: Analysis and Decision Model](https://mlsanity.com/paper/2408.02920/) (2024)
- [FedEval-LLM: Federated Evaluation of Large Language Models on Downstream Tasks with Collective Wisdom](https://mlsanity.com/paper/2404.12273/) (2024)
- [Review of Mathematical Optimization in Federated Learning](https://mlsanity.com/paper/2412.01630/) (2024)
- [A Survey on Foundation Models for Personalized Federated Intelligence](https://mlsanity.com/paper/2505.06907/) (2025)
- [Federated Domain-Specific Knowledge Transfer on Large Language Models Using Synthetic Data](https://mlsanity.com/paper/2405.14212/) (2024)
- [Efficient Federated Learning with Heterogeneous Data and Adaptive Dropout](https://mlsanity.com/paper/2507.10430/) (2025)
- [Foundational Models and Federated Learning: Survey, Taxonomy, Challenges and Practical Insights](https://mlsanity.com/paper/2509.05142/) (2025)
- [A Survey of Personalized Federated Foundation Models for Privacy-Preserving Recommendation](https://mlsanity.com/paper/2506.11563/) (2025)
- [Large-Small Model Collaborative Framework for Federated Continual Learning](https://mlsanity.com/paper/2508.09489/) (2025)
- [A Scenario-Oriented Survey of Federated Recommender Systems: Techniques, Challenges, and Future Directions](https://mlsanity.com/paper/2508.19620/) (2025)
- [FedGRPO: Privately Optimizing Foundation Models with Group-Relative Rewards from Domain Client](https://mlsanity.com/paper/2602.12014/) (2026)
- [FedProxy: Federated Fine-Tuning of LLMs via Proxy SLMs and Heterogeneity-Aware Fusion](https://mlsanity.com/paper/2604.19015/) (2026)

## Abstract (arXiv)

Foundation Models (FMs) such as GPT-4 encoded with vast knowledge and powerful emergent abilities have achieved remarkable success in various natural language processing and computer vision tasks. Grounding FMs by adapting them to domain-specific tasks or augmenting them with domain-specific knowledge enables us to exploit the full potential of FMs. However, grounding FMs faces several challenges, stemming primarily from constrained computing resources, data privacy, model heterogeneity, and model ownership. Federated Transfer Learning (FTL), the combination of federated learning and transfer learning, provides promising solutions to address these challenges. In recent years, the need for grounding FMs leveraging FTL, coined FTL-FM, has arisen strongly in both academia and industry. Motivated by the strong growth in FTL-FM research and the potential impact of FTL-FM on industrial applications, we propose an FTL-FM framework that formulates problems of grounding FMs in the federated learning setting, construct a detailed taxonomy based on the FTL-FM framework to categorize state-of-the-art FTL-FM works, and comprehensively overview FTL-FM works based on the proposed taxonomy. We also establish correspondences between FTL-FM and conventional phases of adapting FM so that FM practitioners can align their research works with FTL-FM. In addition, we overview advanced efficiency-improving and privacy-preserving techniques because efficiency and privacy are critical concerns in FTL-FM. Last, we discuss opportunities and future research directions of FTL-FM.