mlsanityData to 7 Oct 2026

Federated Prompt-Tuning with Heterogeneous and Incomplete Multimodal Client Data

arXiv 2602.07081 ↗First posted 6 Feb 2026cs.MM cs.AI cs.CVcited by 3

Not on the map. Why?

Abstract (arXiv)

This paper introduces a generalized federated prompt-tuning framework for practical scenarios where local datasets are multi-modal and exhibit different distributional patterns of missing features at the input level. The proposed framework bridges the gap between federated learning and multi-modal prompt-tuning which have traditionally focused on either uni-modal or centralized data. A key challenge in this setting arises from the lack of semantic alignment between prompt instructions that encode similar distributional patterns of missing data across different clients. To address this, our framework introduces specialized client-tuning and server-aggregation designs that simultaneously optimize, align, and aggregate prompt-tuning instructions across clients and data modalities. This allows prompt instructions to complement one another and be combined effectively. Extensive evaluations on diverse multimodal benchmark datasets demonstrate that our work consistently outperforms state-of-the-art (SOTA) baselines.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 39 of its 61 references that are in the map. First posted on arXiv 6 Feb 2026.

Builds on

39 of 61 references.

References · 39

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  8. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  9. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  10. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  11. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  12. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
27 more
  1. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  2. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
  3. 2022MaPLe: Multi-modal Prompt Learningcited by 685
  4. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  5. 2020Personalized Federated Learning with Moreau Envelopescited by 612
  6. 2019Bayesian Nonparametric Federated Learning of Neural Networkscited by 385
  7. 2017Gated Multimodal Units for Information Fusioncited by 244
  8. 2022PromptFL: Let Federated Participants Cooperatively Learn Prompts Instead of Models -- Federated Learning in Age of Foundation Modelcited by 133
  9. 2023Multimodal Prompting with Missing Modalities for Visual Recognitioncited by 121
  10. 2022FedPrompt: Communication-Efficient and Privacy Preserving Prompt Tuning in Federated Learningcited by 86
  11. 2022Heterogeneous Ensemble Knowledge Transfer for Training Large Models in Federated Learningcited by 81
  12. 2020Deep Partial Multi-View Learningcited by 71
  13. 2022Where to Begin? On the Impact of Pre-Training and Initialization in Federated Learningcited by 71
  14. 2023FedCLIP: Fast Generalization and Personalization for CLIP in Federated Learningcited by 70
  15. 2023Efficient Model Personalization in Federated Learning via Client-Specific Prompt Generationcited by 54
  16. 2022Geometric Multimodal Contrastive Representation Learningcited by 40
  17. 2022FedTune: A Deep Dive into Efficient Federated Fine-Tuning with Pre-trained Transformerscited by 33
  18. 2019Statistical Model Aggregation via Parameter Matchingcited by 17
  19. 2024FedMAC: Tackling Partial-Modality Missing in Federated Learning with Cross-Modal Aggregation and Contrastive Regularizationcited by 16
  20. 2022Prefix Conditioning Unifies Language and Label Supervisioncited by 13
  21. 2024Incentives in Private Collaborative Machine Learningcited by 10
  22. 2025Probabilistic Federated Prompt-Tuning with Non-IID and Imbalanced Datacited by 9
  23. 2023Federated Learning of Models Pre-Trained on Different Features with Consensus Graphscited by 8
  24. 2022FedDRL: Deep Reinforcement Learning-based Adaptive Aggregation for Non-IID Data in Federated Learningcited by 7
  25. 2023CADIS: Handling Cluster-skewed Non-IID Data in Federated Learning with Clustered Aggregation and Knowledge DIStilled Regularizationcited by 5
  26. 2023Personalized Federated Domain Adaptation for Item-to-Item Recommendationcited by 4
  27. 2022Communication-Efficient and Privacy-Preserving Feature-based Federated Transfer Learningcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Conditional Imputation for Within-Modality Missingness in Multi-Modal Federated Learningcited by 0
  2. 2026Boosting Multimodal Federated Learning via Chained Modality Optimizationcited by 0
  3. 2026FedSocket: Recipient-Executable Knowledge Exchange for Heterogeneous Multimodal Federated Learningcited by 0