mlsanityData to 14 Oct 2026

Ten Challenging Problems in Federated Foundation Models

arXiv 2502.12176 ↗First posted 14 Feb 2025cs.LG cs.AIcited by 20

On the map: LLM safety and security → Federated unlearning and poisoning

Abstract (arXiv)

Federated Foundation Models (FedFMs) represent a distributed learning paradigm that fuses general competences of foundation models as well as privacy-preserving capabilities of federated learning. This combination allows the large foundation models and the small local domain models at the remote clients to learn from each other in a teacher-student learning setting. This paper provides a comprehensive summary of the ten challenging problems inherent in FedFMs, encompassing foundational theory, utilization of private data, continual learning, unlearning, Non-IID and graph data, bidirectional knowledge transfer, incentive mechanism design, game mechanism design, model watermarking, and efficiency. The ten challenging problems manifest in five pivotal aspects: ``Foundational Theory," which aims to establish a coherent and unifying theoretical framework for FedFMs. ``Data," addressing the difficulties in leveraging domain-specific knowledge from private data while maintaining privacy; ``Heterogeneity," examining variations in data, model, and computational resources across clients; ``Security and Privacy," focusing on defenses against malicious attacks and model theft; and ``Efficiency," highlighting the need for improvements in training, communication, and parameter efficiency. For each problem, we offer a clear mathematical definition on the objective function, analyze existing methods, and discuss the key challenges and potential solutions. This in-depth exploration aims to advance the theoretical foundations of FedFMs, guide practical implementations, and inspire future research to overcome these obstacles, thereby enabling the robust, efficient, and privacy-preserving FedFMs in various real-world applications.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Placed in Federated unlearning and poisoning, one of 12 directions in LLM safety and security. Builds on 140 of its 238 references that are in the map. First posted on arXiv 14 Feb 2025.

Builds on

140 of 238 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024Towards Privacy-Guaranteed Label Unlearning in Vertical Federated Learning: Few-Shot Forgetting without Disclosure14 Oct
  2. 2024Unlearning during Learning: An Efficient Federated Machine Unlearning Method24 May
  3. 2024Ferrari: Federated Feature Unlearning via Optimizing Feature Sensitivity23 May
  4. 2024Federated Unlearning: a Perspective of Stability and Fairness2 Feb
  5. 2023Who Leaked the Model? Tracking IP Infringers in Accountable Federated Learning6 Dec
  6. 2023A Survey on Federated Unlearning: Challenges, Methods, and Future Directions31 Oct
  7. 2023FedCIP: Federated Client Intellectual Property Protection with Traitor Tracking2 Jun
  8. 2022SIFU: Sequential Informed Federated Unlearning for Efficient and Provable Client Unlearning in Federated Optimization21 Nov
  9. 2022FedTracker: Furnishing Ownership Verification and Traceability for Federated Learning Model14 Nov
  10. 2022VeriFi: Towards Verifiable Federated Unlearning25 May
  11. 2022The Right to be Forgotten in Federated Learning: An Efficient Realization with Rapid Retraining14 Mar
  12. 2021Federated Unlearning via Class-Discriminative Pruning22 Oct

Other references · 128

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  7. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  8. 2016Learning without Forgettingcited by 2,496
  9. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
  10. 2019Deep Leakage from Gradientscited by 1,021
  11. 2019Towards Federated Learning at Scale: System Designcited by 832
  12. 2018How To Backdoor Federated Learningcited by 765
38 more
  1. 2022Quantifying Memorization Across Neural Language Modelscited by 763
  2. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
  3. 2020Inverting Gradients -- How easy is it to break privacy in federated learning?cited by 712
  4. 2018Byzantine-Robust Distributed Learning: Towards Optimal Statistical Ratescited by 711
  5. 2019Local Model Poisoning Attacks to Byzantine-Robust Federated Learningcited by 493
  6. 2019FedMD: Heterogenous Federated Learning via Model Distillationcited by 459
  7. 2023Learning to Compress Prompts with Gist Tokenscited by 390
  8. 2023A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPTcited by 368
  9. 2018Turning Your Weakness Into a Strength: Watermarking Deep Neural Networks by Backdooringcited by 357
  10. 2020Attack of the Tails: Yes, You Really Can Backdoor Federated Learningcited by 309
  11. 2019On The Power of Curriculum Learning in Training Deep Networkscited by 286
  12. 2020Data Poisoning Attacks Against Federated Learning Systemscited by 275
  13. 2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 251
  14. 2017Embedding Watermarks into Deep Neural Networkscited by 248
  15. 2018Beyond Inferring Class Representatives: User-Level Privacy Leakage From Federated Learningcited by 246
  16. 2019Model Pruning Enables Efficient Federated Learning on Edge Devicescited by 222
  17. 2023Class-Incremental Learning: A Surveycited by 209
  18. 2023Towards Building the Federated GPT: Federated Instruction Tuningcited by 185
  19. 2024GraphRouter: A Graph-based Router for LLM Selectionscited by 165
  20. 2020Federated Continual Learning with Weighted Inter-client Transfercited by 161
  21. 2022Federated Class-Incremental Learningcited by 127
  22. 2020Backdoor Attacks and Countermeasures on Deep Learning: A Comprehensive Reviewcited by 123
  23. 2019[Extended version] Rethinking Deep Neural Network Ownership Verification: Embedding Passports to Defeat Ambiguity Attackscited by 88
  24. 2023Multimodal Federated Learning via Contrastive Representation Ensemblecited by 86
  25. 2023When Foundation Model Meets Federated Learning: Motivations, Challenges, and Future Directionscited by 84
  26. 2023Offsite-Tuning: Transfer Learning without Full Modelcited by 78
  27. 2019Federated Learning for Edge Networks: Resource Optimization and Incentive Mechanismcited by 75
  28. 2024FedBiOT: LLM Local Fine-tuning in Federated Learning without Full Modelcited by 74
  29. 2021Tackling System and Statistical Heterogeneity for Federated Learning with Adaptive Client Samplingcited by 66
  30. 2020Learn to Forget: Machine Unlearning via Neuron Maskingcited by 62
  31. 2020Cost-Effective Federated Learning Designcited by 62
  32. 2023FedDAT: An Approach for Foundation Model Finetuning in Multi-Modal Heterogeneous Federated Learningcited by 60
  33. 2022FedRecover: Recovering from Poisoning Attacks in Federated Learning using Historical Informationcited by 58
  34. 2023Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytescited by 55
  35. 2024Dual-Personalizing Adapter for Federated Foundation Modelscited by 55
  36. 2022Personalized Subgraph Federated Learningcited by 54
  37. 2023Efficient Model Personalization in Federated Learning via Client-Specific Prompt Generationcited by 54
  38. 2023Federated Continual Learning via Knowledge Fusion: A Surveycited by 53

50 most cited of 128 All 128 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 13; 2026: 7.

20 papers.

Sort:

By area:

  1. 2025Federated Deconfounding and Debiasing Learning for Out-of-Distribution Generalizationcited by 4
  2. 2025A Survey on Foundation Models for Personalized Federated Intelligencecited by 3
  3. 2025FedHL: Federated Learning for Heterogeneous Low-Rank Adaptation via Unbiased Aggregationcited by 3
  4. 2025Global Prompt Refinement with Non-Interfering Attention Masking for One-Shot Federated Learningcited by 2
  5. 2025Adaptive Federated LoRA in Heterogeneous Wireless Networks with Independent Samplingcited by 1
  6. 2026EdgeFlowerTune: Evaluating Federated LLM Fine-Tuning Under Realistic Edge System Constraintscited by 1
  7. 2025Robust Federated Learning in Unreliable Wireless Networks: A Client Selection Approachcited by 0
  8. 2025FHBench: Towards Efficient and Personalized Federated Learning for Multimodal Healthcarecited by 0
  9. 2025Local Data Quantity-Aware Weighted Averaging for Federated Learning with Dishonest Clientscited by 0
  10. 2025Voronoi-grid-based Pareto Front Learning and Its Application to Collaborative Federated Learningcited by 0
  11. 2025DAG-AFL:Directed Acyclic Graph-based Asynchronous Federated Learningcited by 0
  12. 2025Large-Small Model Collaborative Framework for Federated Continual Learningcited by 0
8 more
  1. 2025Gains: Fine-grained Federated Domain Adaptation in Open Setcited by 0
  2. 2025Enhanced Federated Deep Multi-View Clustering under Uncertainty Scenariocited by 0
  3. 2026FedGRPO: Privately Optimizing Foundation Models with Group-Relative Rewards from Domain Clientcited by 0
  4. 2026HiLoRA: Hierarchical Low-Rank Adaptation for Personalized Federated Learningcited by 0
  5. 2026FedProxy: Federated Fine-Tuning of LLMs via Proxy SLMs and Heterogeneity-Aware Fusioncited by 0
  6. 2026Silent Failures in Federated Personalization of Foundation Modelscited by 0
  7. 2026Fisher-Routed Mixture of Experts for Federated Class-Incremental Learningcited by 0
  8. 2026Out-of-Distribution Federated Distillation with Domain-Aware Proxycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Foundation Models in Federated Learning: Assessing Backdoor Vulnerabilities
  2. 2023A Survey of What to Share in Federated Learning: Perspectives on Model Utility, Privacy Leakage, and Communication Efficiency
  3. 2024Securing Federated Learning against Backdoor Threats with Foundation Model Integration
  4. 2025Bringing Multi-Modal Multi-Task Federated Foundation Models to Education Domain: Prospects and Challenges
  5. 2024Federated Learning: Attacks, Defenses, Opportunities, and Challenges
  6. 2023Selective Knowledge Sharing for Privacy-Preserving Federated Distillation without A Good Teacher
  7. 2024FedSiKD: Clients Similarity and Knowledge Distillation: Addressing Non-i.i.d. and Constraints in Federated Learning
  8. 2023A Survey on Vulnerability of Federated Learning: A Learning Algorithm Perspective