# Ten Challenging Problems in Federated Foundation Models

arXiv: https://arxiv.org/abs/2502.12176 · HTML: https://mlsanity.com/paper/2502.12176/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Placed in Federated unlearning and poisoning, one of 12 directions in LLM safety and security.
- Builds on 140 of its 238 references that are in the map.
- First posted on arXiv 14 Feb 2025.

## Placed in

- LLM safety and security / [Federated unlearning and poisoning](https://mlsanity.com/safety/federated-unlearning-and-poisoning/)

## Builds on (most cited first, 30 of 140)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [Understanding Black-box Predictions via Influence Functions](https://mlsanity.com/paper/1703.04730/) (2017)
- [Deep Leakage from Gradients](https://mlsanity.com/paper/1906.08935/) (2019)
- [Towards Federated Learning at Scale: System Design](https://mlsanity.com/paper/1902.01046/) (2019)
- [How To Backdoor Federated Learning](https://mlsanity.com/paper/1807.00459/) (2018)
- [Quantifying Memorization Across Neural Language Models](https://mlsanity.com/paper/2202.07646/) (2022)
- [Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes](https://mlsanity.com/paper/2305.02301/) (2023)
- [Inverting Gradients -- How easy is it to break privacy in federated learning?](https://mlsanity.com/paper/2003.14053/) (2020)
- [Byzantine-Robust Distributed Learning: Towards Optimal Statistical Rates](https://mlsanity.com/paper/1803.01498/) (2018)
- [Local Model Poisoning Attacks to Byzantine-Robust Federated Learning](https://mlsanity.com/paper/1911.11815/) (2019)
- [FedMD: Heterogenous Federated Learning via Model Distillation](https://mlsanity.com/paper/1910.03581/) (2019)
- [Learning to Compress Prompts with Gist Tokens](https://mlsanity.com/paper/2304.08467/) (2023)
- [A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPT](https://mlsanity.com/paper/2302.09419/) (2023)
- [Turning Your Weakness Into a Strength: Watermarking Deep Neural Networks by Backdooring](https://mlsanity.com/paper/1802.04633/) (2018)
- [Attack of the Tails: Yes, You Really Can Backdoor Federated Learning](https://mlsanity.com/paper/2007.05084/) (2020)
- [On The Power of Curriculum Learning in Training Deep Networks](https://mlsanity.com/paper/1904.03626/) (2019)
- [Data Poisoning Attacks Against Federated Learning Systems](https://mlsanity.com/paper/2007.08432/) (2020)
- [DynaBERT: Dynamic BERT with Adaptive Width and Depth](https://mlsanity.com/paper/2004.04037/) (2020)
- [Embedding Watermarks into Deep Neural Networks](https://mlsanity.com/paper/1701.04082/) (2017)
- [Beyond Inferring Class Representatives: User-Level Privacy Leakage From Federated Learning](https://mlsanity.com/paper/1812.00535/) (2018)
- [Model Pruning Enables Efficient Federated Learning on Edge Devices](https://mlsanity.com/paper/1909.12326/) (2019)
- [Class-Incremental Learning: A Survey](https://mlsanity.com/paper/2302.03648/) (2023)
- [Towards Building the Federated GPT: Federated Instruction Tuning](https://mlsanity.com/paper/2305.05644/) (2023)

## Built on by (most cited first, 20 of 20)

- [Federated Deconfounding and Debiasing Learning for Out-of-Distribution Generalization](https://mlsanity.com/paper/2505.04979/) (2025)
- [A Survey on Foundation Models for Personalized Federated Intelligence](https://mlsanity.com/paper/2505.06907/) (2025)
- [FedHL: Federated Learning for Heterogeneous Low-Rank Adaptation via Unbiased Aggregation](https://mlsanity.com/paper/2505.18494/) (2025)
- [Global Prompt Refinement with Non-Interfering Attention Masking for One-Shot Federated Learning](https://mlsanity.com/paper/2509.22700/) (2025)
- [Adaptive Federated LoRA in Heterogeneous Wireless Networks with Independent Sampling](https://mlsanity.com/paper/2505.23555/) (2025)
- [EdgeFlowerTune: Evaluating Federated LLM Fine-Tuning Under Realistic Edge System Constraints](https://mlsanity.com/paper/2605.08636/) (2026)
- [Robust Federated Learning in Unreliable Wireless Networks: A Client Selection Approach](https://mlsanity.com/paper/2502.17260/) (2025)
- [FHBench: Towards Efficient and Personalized Federated Learning for Multimodal Healthcare](https://mlsanity.com/paper/2504.10817/) (2025)
- [Local Data Quantity-Aware Weighted Averaging for Federated Learning with Dishonest Clients](https://mlsanity.com/paper/2504.12577/) (2025)
- [Voronoi-grid-based Pareto Front Learning and Its Application to Collaborative Federated Learning](https://mlsanity.com/paper/2505.20648/) (2025)
- [DAG-AFL:Directed Acyclic Graph-based Asynchronous Federated Learning](https://mlsanity.com/paper/2507.20571/) (2025)
- [Large-Small Model Collaborative Framework for Federated Continual Learning](https://mlsanity.com/paper/2508.09489/) (2025)
- [Gains: Fine-grained Federated Domain Adaptation in Open Set](https://mlsanity.com/paper/2510.15967/) (2025)
- [Enhanced Federated Deep Multi-View Clustering under Uncertainty Scenario](https://mlsanity.com/paper/2511.17631/) (2025)
- [FedGRPO: Privately Optimizing Foundation Models with Group-Relative Rewards from Domain Client](https://mlsanity.com/paper/2602.12014/) (2026)
- [HiLoRA: Hierarchical Low-Rank Adaptation for Personalized Federated Learning](https://mlsanity.com/paper/2603.02785/) (2026)
- [FedProxy: Federated Fine-Tuning of LLMs via Proxy SLMs and Heterogeneity-Aware Fusion](https://mlsanity.com/paper/2604.19015/) (2026)
- [Silent Failures in Federated Personalization of Foundation Models](https://mlsanity.com/paper/2606.00947/) (2026)
- [Fisher-Routed Mixture of Experts for Federated Class-Incremental Learning](https://mlsanity.com/paper/2606.28835/) (2026)
- [Out-of-Distribution Federated Distillation with Domain-Aware Proxy](https://mlsanity.com/paper/2608.08525/) (2026)

## Abstract (arXiv)

Federated Foundation Models (FedFMs) represent a distributed learning paradigm that fuses general competences of foundation models as well as privacy-preserving capabilities of federated learning. This combination allows the large foundation models and the small local domain models at the remote clients to learn from each other in a teacher-student learning setting. This paper provides a comprehensive summary of the ten challenging problems inherent in FedFMs, encompassing foundational theory, utilization of private data, continual learning, unlearning, Non-IID and graph data, bidirectional knowledge transfer, incentive mechanism design, game mechanism design, model watermarking, and efficiency. The ten challenging problems manifest in five pivotal aspects: ``Foundational Theory," which aims to establish a coherent and unifying theoretical framework for FedFMs. ``Data," addressing the difficulties in leveraging domain-specific knowledge from private data while maintaining privacy; ``Heterogeneity," examining variations in data, model, and computational resources across clients; ``Security and Privacy," focusing on defenses against malicious attacks and model theft; and ``Efficiency," highlighting the need for improvements in training, communication, and parameter efficiency. For each problem, we offer a clear mathematical definition on the objective function, analyze existing methods, and discuss the key challenges and potential solutions. This in-depth exploration aims to advance the theoretical foundations of FedFMs, guide practical implementations, and inspire future research to overcome these obstacles, thereby enabling the robust, efficient, and privacy-preserving FedFMs in various real-world applications.