mlsanityData to 14 Oct 2026

ZooPFL: Exploring Black-box Foundation Models for Personalized Federated Learning

arXiv 2310.05143 ↗First posted 8 Oct 2023cs.AI cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

When personalized federated learning (FL) meets large foundation models, new challenges arise from various limitations in resources. In addition to typical limitations such as data, computation, and communication costs, access to the models is also often limited. This paper endeavors to solve both the challenges of limited resources and personalization. i.e., distribution shifts between clients. To do so, we propose a method named ZOOPFL that uses Zeroth-Order Optimization for Personalized Federated Learning. ZOOPFL avoids direct interference with the foundation models and instead learns to adapt its inputs through zeroth-order optimization. In addition, we employ simple yet effective linear projections to remap its predictions for personalization. To reduce the computation costs and enhance personalization, we propose input surgery to incorporate an auto-encoder with low-dimensional and client-specific embeddings. We provide theoretical support for ZOOPFL to analyze its convergence. Extensive empirical experiments on computer vision and natural language processing tasks using popular foundation models demonstrate its effectiveness for FL on black-box foundation models.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 76 of its 121 references that are in the map. First posted on arXiv 8 Oct 2023.

Builds on

76 of 121 references.

References · 76

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  8. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  9. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  10. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  11. 2021Learning to Prompt for Vision-Language Modelscited by 2,281
  12. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
38 more
  1. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  2. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  3. 2018A Survey Of Methods For Explaining Black Box Modelscited by 973
  4. 2018Neural Network Acceptability Judgmentscited by 958
  5. 2021Learning to Prompt for Continual Learningcited by 944
  6. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  7. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 724
  8. 2018Recognition in Terra Incognitacited by 658
  9. 2016Stealing Machine Learning Models via Prediction APIscited by 619
  10. 2023A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPTcited by 619
  11. 2021FedBN: Federated Learning on Non-IID Features via Local Batch Normalizationcited by 571
  12. 2019Robust and Communication-Efficient Federated Learning from Non-IID Datacited by 419
  13. 2019Bayesian Nonparametric Federated Learning of Neural Networkscited by 387
  14. 2013Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Textscited by 310
  15. 2022Black-Box Tuning for Language-Model-as-a-Servicecited by 256
  16. 2021Design Guidelines for Prompt Engineering Text-to-Image Generative Modelscited by 218
  17. 2021FedBABU: Towards Enhanced Representation for Federated Image Classificationcited by 182
  18. 2022FedDC: Federated Learning with Non-IID Data via Local Drift Decoupling and Correctioncited by 181
  19. 2020A Primer on Zeroth-Order Optimization in Signal Processing and Machine Learningcited by 162
  20. 2023FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learningcited by 149
  21. 2022Generalized Federated Learning via Sharpness Aware Minimizationcited by 144
  22. 2018AutoZOOM: Autoencoder-based Zeroth Order Optimization Method for Attacking Black-box Neural Networkscited by 137
  23. 2022PromptFL: Let Federated Participants Cooperatively Learn Prompts Instead of Models -- Federated Learning in Age of Foundation Modelcited by 134
  24. 2022Federated Learning with Partial Model Personalizationcited by 114
  25. 2018Zeroth-Order Stochastic Variance Reduction for Nonconvex Optimizationcited by 111
  26. 2022Vertical Federated Learning: Concepts, Advances and Challengescited by 108
  27. 2023Personalized Federated Learning with Feature Alignment and Classifier Collaborationcited by 97
  28. 2022FedPrompt: Communication-Efficient and Privacy Preserving Prompt Tuning in Federated Learningcited by 91
  29. 2022Survey on Federated Learning Threats: concepts, taxonomy on attacks and defences, experimental study and challengescited by 84
  30. 2023When Foundation Model Meets Federated Learning: Motivations, Challenges, and Future Directionscited by 84
  31. 2020Transfer Learning without Knowing: Reprogramming Black-box Machine Learning Models with Scarce Data and Limited Resourcescited by 83
  32. 2023Offsite-Tuning: Transfer Learning without Full Modelcited by 78
  33. 2022On the Importance and Applicability of Pre-Training for Federated Learningcited by 71
  34. 2023FedCLIP: Fast Generalization and Personalization for CLIP in Federated Learningcited by 70
  35. 2022Model Reprogramming: Resource-Efficient Cross-Domain Machine Learningcited by 64
  36. 2017Stochastic Zeroth-order Optimization in High Dimensionscited by 63
  37. 2022Communication-Efficient Stochastic Zeroth-Order Optimization for Federated Learningcited by 59
  38. 2020DarkneTZ: Towards Model Privacy at the Edge using Trusted Execution Environmentscited by 58

50 most cited of 76 All 76 (JSON)

Built on by

Cited by year

Citing papers per year: 2024: 4; 2025: 5.

9 papers.

  1. 2024Federated Large Language Models: Current Progress and Future Directionscited by 27
  2. 2025Ten Challenging Problems in Federated Foundation Modelscited by 20
  3. 2024On Catastrophic Inheritance of Large Foundation Modelscited by 15
  4. 2024Synergizing Foundation Models and Federated Learning: A Surveycited by 14
  5. 2024PEFT-as-an-Attack! Jailbreaking Language Models during Federated Parameter-Efficient Fine-Tuningcited by 5
  6. 2025Generalized and Personalized Federated Learning with Black-Box Foundation Models via Orthogonal Transformationscited by 1
  7. 2025Foundational Models and Federated Learning: Survey, Taxonomy, Challenges and Practical Insightscited by 1
  8. 2025A Randomized Zeroth-Order Hierarchical Framework for Heterogeneous Federated Learningcited by 0
  9. 2025Federated Foundation Language Model Post-Training Should Focus on Open-Source Modelscited by 0