# Global Prompt Refinement with Non-Interfering Attention Masking for One-Shot Federated Learning

arXiv: https://arxiv.org/abs/2509.22700 · HTML: https://mlsanity.com/paper/2509.22700/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 80 references that are in the map.
- First posted on arXiv 21 Sep 2025.

## Builds on (most cited first, 30 of 41)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [Federated Learning: Challenges, Methods, and Future Directions](https://mlsanity.com/paper/1908.07873/) (2019)
- [Moment Matching for Multi-Source Domain Adaptation](https://mlsanity.com/paper/1812.01754/) (2018)
- [PromptFL: Let Federated Participants Cooperatively Learn Prompts Instead of Models -- Federated Learning in Age of Foundation Model](https://mlsanity.com/paper/2208.11625/) (2022)
- [Federated Class-Incremental Learning](https://mlsanity.com/paper/2203.11473/) (2022)
- [DENSE: Data-Free One-Shot Federated Learning](https://mlsanity.com/paper/2112.12371/) (2021)
- [Federated Learning for Generalization, Robustness, Fairness: A Survey and Benchmark](https://mlsanity.com/paper/2311.06750/) (2023)
- [FedPrompt: Communication-Efficient and Privacy Preserving Prompt Tuning in Federated Learning](https://mlsanity.com/paper/2208.12268/) (2022)
- [TCP:Textual-based Class-aware Prompt tuning for Visual-Language Model](https://mlsanity.com/paper/2311.18231/) (2023)
- [Read-only Prompt Optimization for Vision-Language Few-shot Learning](https://mlsanity.com/paper/2308.14960/) (2023)
- [Efficient Model Personalization in Federated Learning via Client-Specific Prompt Generation](https://mlsanity.com/paper/2308.15367/) (2023)
- [Global and Local Prompts Cooperation via Optimal Transport for Federated Learning](https://mlsanity.com/paper/2403.00041/) (2024)
- [Voice-Face Homogeneity Tells Deepfake](https://mlsanity.com/paper/2203.02195/) (2022)
- [Advances and Open Challenges in Federated Foundation Models](https://mlsanity.com/paper/2404.15381/) (2024)
- [Cross-Silo Prototypical Calibration for Federated Learning with Non-IID Data](https://mlsanity.com/paper/2308.03457/) (2023)
- [Exploring One-shot Semi-supervised Federated Learning with A Pre-trained Diffusion Model](https://mlsanity.com/paper/2305.04063/) (2023)
- [DiPrompT: Disentangled Prompt Tuning for Multiple Latent Domain Generalization in Federated Learning](https://mlsanity.com/paper/2403.08506/) (2024)
- [Federated Adaptive Prompt Tuning for Multi-Domain Collaborative Learning](https://mlsanity.com/paper/2211.07864/) (2022)
- [Harmonizing Generalization and Personalization in Federated Prompt Learning](https://mlsanity.com/paper/2405.09771/) (2024)
- [Federated Learning from Vision-Language Foundation Models: Theoretical Analysis and Method](https://mlsanity.com/paper/2409.19610/) (2024)
- [FedCross: Towards Accurate Federated Learning via Multi-Model Cross-Aggregation](https://mlsanity.com/paper/2210.08285/) (2022)
- [Federated Model Heterogeneous Matryoshka Representation Learning](https://mlsanity.com/paper/2406.00488/) (2024)
- [Exploiting Multi-Label Correlation in Label Distribution Learning](https://mlsanity.com/paper/2308.01742/) (2023)
- [Unlocking the Potential of Prompt-Tuning in Bridging Generalized and Personalized Federated Learning](https://mlsanity.com/paper/2310.18285/) (2023)
- [Ten Challenging Problems in Federated Foundation Models](https://mlsanity.com/paper/2502.12176/) (2025)
- [FedDEO: Description-Enhanced One-Shot Federated Learning with Diffusion Models](https://mlsanity.com/paper/2407.19953/) (2024)
- [Revisiting Ensembling in One-Shot Federated Learning](https://mlsanity.com/paper/2411.07182/) (2024)

## Built on by (most cited first, 2 of 2)

- [From Selection to Scheduling: Federated Geometry-Aware Correction Makes Exemplar Replay Work Better under Continual Dynamic Heterogeneity](https://mlsanity.com/paper/2604.08617/) (2026)
- [Enhancing Self-Supervised Talking Head Forgery Detection via a Training-Free Dual-System Framework](https://mlsanity.com/paper/2605.03390/) (2026)

## Abstract (arXiv)

Federated Prompt Learning (FPL) enables communication-efficient adaptation by tuning lightweight prompts on top of frozen pre-trained models. Existing FPL methods typically rely on global information, which is only available after the second training round, to facilitate collaboration among client models. Therefore, they are inherently dependent on multi-round communication to fully exhibit their strengths. Moreover, existing one-shot federated learning methods typically focus on fitting seen tasks, but lack cross-task generalization. To bridge this gap, we propose the Global Prompt Refinement with Non-Interfering Attention Masking (GPR-NIAM) method for one-shot FPL. The core idea is to design a masking mechanism that restricts excessive interaction between the original text embeddings and the learnable prompt embeddings. GPR-NIAM achieves this through the collaboration of two key modules. Firstly, the attention isolation module suppresses attention from the learnable prompt tokens to the original text tokens, and reweights the reverse attention which preserves generalization across tasks. Secondly, the cross-silo collaborative refinement module integrates decentralized visual knowledge into a unified base and calibrates the global prompt through multi-source cross-modal knowledge alignment, further mitigating the inconsistency caused by data heterogeneity. Extensive experiments conducted on ten benchmark datasets under two tasks show that GPR-NIAM outperforms eight state-of-the-art methods in both class-level and domain-level generalization.