mlsanityData to 14 Oct 2026

FedPIA -- Permuting and Integrating Adapters leveraging Wasserstein Barycenters for Finetuning Foundation Models in Multi-Modal Federated Learning

arXiv 2412.14424 ↗First posted 19 Dec 2024cs.CV cs.AI cs.LGcited by 6

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Large Vision-Language Models typically require large text and image datasets for effective fine-tuning. However, collecting data from various sites, especially in healthcare, is challenging due to strict privacy regulations. An alternative is to fine-tune these models on end-user devices, such as in medical clinics, without sending data to a server. These local clients typically have limited computing power and small datasets, which are not enough for fully fine-tuning large VLMs on their own. A naive solution to these scenarios is to leverage parameter-efficient fine-tuning (PEFT) strategies and apply federated learning (FL) algorithms to combine the learned adapter weights, thereby respecting the resource limitations and data privacy. However, this approach does not fully leverage the knowledge from multiple adapters trained on diverse data distributions and for diverse tasks. The adapters are adversely impacted by data heterogeneity and task heterogeneity across clients resulting in suboptimal convergence. To this end, we propose a novel framework called FedPIA that improves upon the naive combinations of FL and PEFT by introducing Permutation and Integration of the local Adapters in the server and global Adapters in the clients exploiting Wasserstein barycenters for improved blending of client-specific and client-agnostic knowledge. This layerwise permutation helps to bridge the gap in the parameter space of local and global adapters before integration. We conduct over 2000 client-level experiments utilizing 48 medical image datasets across five different medical vision-language FL task settings encompassing visual question answering as well as image and report-based multi-label disease detection. Our experiments involving diverse client settings, ten different modalities, and two VLM backbones demonstrate that FedPIA consistently outperforms the state-of-the-art PEFT-FL baselines.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 21 of its 54 references that are in the map. First posted on arXiv 19 Dec 2024.

Builds on

21 of 54 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024F3OCUS -- Federated Finetuning of Vision-Language Foundation Models with Optimal Client Layer Updating Strategy via Multi-objective Meta-Heuristics17 Nov
  2. 2024FLoRA: Enhancing Vision-Language Models with Parameter-Efficient Federated Learning12 Apr

Other references · 19

  1. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  2. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  3. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 788
  4. 2021Federated Learning Based on Dynamic Regularizationcited by 562
  5. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 556
  6. 2022Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuningcited by 294
  7. 2020Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNscited by 104
  8. 2023Multimodal Federated Learning via Contrastive Representation Ensemblecited by 86
  9. 2023When Foundation Model Meets Federated Learning: Motivations, Challenges, and Future Directionscited by 84
  10. 2023FedCLIP: Fast Generalization and Personalization for CLIP in Federated Learningcited by 70
  11. 2023Federated Foundation Models: Privacy-Preserving and Collaborative Learning for Large Modelscited by 50
  12. 2023Strong Baselines for Parameter Efficient Few-Shot Fine-tuningcited by 37
7 more
  1. 2022FedTune: A Deep Dive into Efficient Federated Fine-Tuning with Pre-trained Transformerscited by 33
  2. 2022Federated Adaptive Prompt Tuning for Multi-Domain Collaborative Learningcited by 29
  3. 2023Visual Prompt Based Personalized Federated Learningcited by 16
  4. 2022Wasserstein Barycenter-based Model Fusion and Linear Mode Connectivity of Neural Networkscited by 15
  5. 2024Examining Modality Incongruity in Multimodal Federated Learning for Medical Vision and Language-based Disease Detectioncited by 8
  6. 2024Feasibility of Federated Learning from Client Databases with Different Brain Diseases and MRI Modalitiescited by 5
  7. 2024Open-Vocabulary Federated Learning with Multimodal Prototypingcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 2.

6 papers.

  1. 2025Vision Foundation Models in Medical Image Analysis: Advances and Challengescited by 9
  2. 2025A Survey on Foundation Models for Personalized Federated Intelligencecited by 3
  3. 2025MCAT: Visual Query-Based Localization of Standard Anatomical Clips in Fetal Ultrasound Videos Using Multi-Tier Class-Aware Token Transformercited by 2
  4. 2025FedDEAP: Adaptive Dual-Prompt Tuning for Multi-Domain Federated Learningcited by 1
  5. 2026FedEU: Evidential Uncertainty-Driven Federated Fine-Tuning of Vision Foundation Models for Remote Sensing Image Segmentationcited by 0
  6. 2026Prototype-guided Bilateral Alignment Multimodal Federated Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025FedVLMBench: Benchmarking Federated Fine-Tuning of Vision-Language Models
  2. 2025FedVLM: Scalable Personalized Vision-Language Models through Federated Learning
  3. 2025pFedMMA: Personalized Federated Fine-Tuning with Multi-Modal Adapter for Vision-Language Models
  4. 2023FedDAT: An Approach for Foundation Model Finetuning in Multi-Modal Heterogeneous Federated Learning
  5. 2026Decoupled Training with Local Reinforcement Fine-Tuning in Federated Learning
  6. 2026Replacing Parameters with Preferences: Federated Alignment of Heterogeneous Vision-Language Models
  7. 2026Replacing Parameters with Preferences: Federated Alignment of Heterogeneous Vision-Language Models
  8. 2025FedCoT: Communication-Efficient Federated Reasoning Enhancement for Large Language Models