mlsanityData to 14 Oct 2026

FedAPT: Federated Adversarial Prompt Tuning for Vision-Language Models

arXiv 2509.06992 ↗First posted 3 Sep 2025cs.CVcited by 1

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

also in 1 more
Abstract (arXiv)

Federated Prompt Tuning (FPT) is an efficient method for cross-client collaborative fine-tuning of large Vision-Language Models (VLMs). However, models tuned using FPT are vulnerable to adversarial attacks, leading to misclassification in downstream tasks. In this work, we introduce Federated Adversarial Prompt Tuning (FedAPT), a novel method designed to enhance the adversarial robustness of FPT. We identify a key issue in FedAPT under non-independent and identically distributed (non-IID) settings: a textitclass information gap between clients and the global model. Clients rely solely on limited local label information to generate adversarial samples for training, while the global model must defend against adversarial attacks from global labels. To address this issue, we propose a class-aware prompt generator that generates visual prompts from text prompts. This generator is guided by a Global Label Embedding (serving as a ``beacon") which encodes cross-client label information to create more globally-aligned visual prompts. Additionally, we propose a cross-layer generator sharing strategy to enhance prompt coupling across different layers of the model, further boosting adversarial robustness. Extensive experiments on multiple image classification datasets demonstrate the superiority of FedAPT in improving adversarial robustness, outperforming existing methods by a large margin. FedAPT also exhibits exceptional generalization in cross-domain and cross-dataset scenarios, indicating its effectiveness in real-world applications.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 32 of its 43 references that are in the map. First posted on arXiv 3 Sep 2025.

Builds on

32 of 43 references.

References · 32

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  3. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  4. 2013Intriguing properties of neural networkscited by 5,155
  5. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,998
  6. 2013Describing Textures in the Wildcited by 2,125
  7. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  8. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  9. 2022Visual Prompt Tuningcited by 1,681
  10. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  11. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
  12. 2019Natural Adversarial Examplescited by 1,408
20 more
  1. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,013
  2. 2018Improving Transferability of Adversarial Examples with Input Diversitycited by 599
  3. 2021True Few-Shot Learning with Language Modelscited by 344
  4. 2020Torchattacks: A PyTorch Repository for Adversarial Attackscited by 158
  5. 2022PromptFL: Let Federated Participants Cooperatively Learn Prompts Instead of Models -- Federated Learning in Age of Foundation Modelcited by 134
  6. 2022LAS-AT: Adversarial Training with Learnable Attack Strategycited by 95
  7. 2022FedPrompt: Communication-Efficient and Privacy Preserving Prompt Tuning in Federated Learningcited by 91
  8. 2021On the effectiveness of adversarial training against common corruptionscited by 71
  9. 2023Federated Learning of Large Language Models with Parameter-Efficient Prompt Tuning and Adaptive Optimizationcited by 65
  10. 2023Efficient Model Personalization in Federated Learning via Client-Specific Prompt Generationcited by 54
  11. 2024One Prompt Word is Enough to Boost Adversarial Robustness for Pre-trained Vision-Language Modelscited by 48
  12. 2022A Survey of Robust Adversarial Training in Pattern Recognition: Fundamental, Theory, and Methodologiescited by 44
  13. 2023Adversarial Prompt Tuning for Vision-Language Modelscited by 44
  14. 2024Few-Shot Adversarial Prompt Learning on Vision-Language Modelscited by 39
  15. 2022Visual Prompting for Adversarial Robustnesscited by 34
  16. 2024DiPrompT: Disentangled Prompt Tuning for Multiple Latent Domain Generalization in Federated Learningcited by 32
  17. 2023LION: Implicit Vision Prompt Tuningcited by 22
  18. 2023Unlocking the Potential of Prompt-Tuning in Bridging Generalized and Personalized Federated Learningcited by 20
  19. 2023Delving into the Adversarial Robustness of Federated Learningcited by 19
  20. 2021Adversarial training in communication constrained federated learningcited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026What Makes VLMs Robust? Towards Reconciling Robustness and Accuracy in Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025FedMVP: Federated Multimodal Visual Prompt Tuning for Vision-Language Models
  2. 2025FedDEAP: Adaptive Dual-Prompt Tuning for Multi-Domain Federated Learning
  3. 2025FOCoOp: Enhancing Out-of-Distribution Robustness in Federated Prompt Learning for Vision-Language Models
  4. 2025MoAPT: Mixture of Adversarial Prompt Tuning for Vision-Language Models
  5. 2025FedVLMBench: Benchmarking Federated Fine-Tuning of Vision-Language Models
  6. 2023Text-driven Prompt Generation for Vision-Language Models in Federated Learning
  7. 2025An Empirical Study of Federated Prompt Learning for Vision Language Model
  8. 2026FedMPT: Federated Multi-label Prompt Tuning of Vision-Language Models