mlsanityData to 7 Oct 2026

F3OCUS -- Federated Finetuning of Vision-Language Foundation Models with Optimal Client Layer Updating Strategy via Multi-objective Meta-Heuristics

arXiv 2411.11912 ↗First posted 17 Nov 2024cs.CV cs.AI cs.LGcited by 4

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Effective training of large Vision-Language Models (VLMs) on resource-constrained client devices in Federated Learning (FL) requires the usage of parameter-efficient fine-tuning (PEFT) strategies. To this end, we demonstrate the impact of two factors textitviz., client-specific layer importance score that selects the most important VLM layers for fine-tuning and inter-client layer diversity score that encourages diverse layer selection across clients for optimal VLM layer selection. We first theoretically motivate and leverage the principal eigenvalue magnitude of layerwise Neural Tangent Kernels and show its effectiveness as client-specific layer importance score. Next, we propose a novel layer updating strategy dubbed F3OCUS that jointly optimizes the layer importance and diversity factors by employing a data-free, multi-objective, meta-heuristic optimization on the server. We explore 5 different meta-heuristic algorithms and compare their effectiveness for selecting model layers and adapter layers towards PEFT-FL. Furthermore, we release a new MedVQA-FL dataset involving overall 707,962 VQA triplets and 9 modality-specific clients and utilize it to train and evaluate our method. Overall, we conduct more than 10,000 client-level experiments on 6 Vision-Language FL task settings involving 58 medical image datasets and 4 different VLM architectures of varying sizes to demonstrate the effectiveness of the proposed method.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 92 of its 149 references that are in the map. First posted on arXiv 17 Nov 2024.

Builds on

92 of 149 references.

References · 92

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  8. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  9. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  10. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  11. 2022Visual Prompt Tuningcited by 1,674
  12. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
38 more
  1. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,531
  2. 2018On the Spectral Bias of Neural Networkscited by 1,321
  3. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  4. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  5. 2017ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseasescited by 969
  6. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 897
  7. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  8. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
  9. 2021MedMNIST v2 -- A large-scale lightweight benchmark for 2D and 3D biomedical image classificationcited by 682
  10. 2019Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)cited by 657
  11. 2021Federated Learning Based on Dynamic Regularizationcited by 555
  12. 2021Exploiting Shared Representations for Personalized Federated Learningcited by 537
  13. 2019Federated Learning with Personalization Layerscited by 533
  14. 2019The Liver Tumor Segmentation Benchmark (LiTS)cited by 504
  15. 2020Pruning neural networks without any data by iteratively conserving synaptic flowcited by 475
  16. 2020Picking Winning Tickets Before Training by Preserving Gradient Flowcited by 467
  17. 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 467
  18. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
  19. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 444
  20. 2019A large annotated medical image dataset for the development and evaluation of segmentation algorithmscited by 371
  21. 2021FedProto: Federated Prototype Learning across Heterogeneous Clientscited by 349
  22. 2020CHAOS Challenge -- Combined (CT-MR) Healthy Abdominal Organ Segmentationcited by 307
  23. 2019Revealing the Dark Secrets of BERTcited by 300
  24. 2022Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuningcited by 292
  25. 2020COVID-Net: A Tailored Deep Convolutional Neural Network Design for Detection of COVID-19 Cases from Chest X-Ray Imagescited by 264
  26. 2020AdapterDrop: On the Efficiency of Adapters in Transformerscited by 223
  27. 2021Training Neural Networks with Fixed Sparse Maskscited by 212
  28. 2020MedMNIST Classification Decathlon: A Lightweight AutoML Benchmark for Medical Image Analysiscited by 206
  29. 2022Surgical Fine-Tuning Improves Adaptation to Distribution Shiftscited by 205
  30. 2020Personalized Federated Learning with First Order Model Optimizationcited by 201
  31. 2020Can AI help in screening Viral and COVID-19 pneumonia?cited by 200
  32. 2016Skin Lesion Analysis toward Melanoma Detection: A Challenge at the International Symposium on Biomedical Imaging (ISBI) 2016, hosted by the International Skin Imaging Collaboration (ISIC)cited by 197
  33. 2021Evaluating Deep Neural Networks Trained on Clinical Images in Dermatology with the Fitzpatrick 17k Datasetcited by 184
  34. 2023Towards Building the Federated GPT: Federated Instruction Tuningcited by 184
  35. 2024OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLMcited by 182
  36. 2019Towards Understanding the Spectral Bias of Deep Learningcited by 173
  37. 2022Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learningcited by 172
  38. 2018Efficient parametrization of multi-domain deep neural networkscited by 168

50 most cited of 92 All 92 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 1; 2026: 2.

4 papers.

  1. 2024FedPIA -- Permuting and Integrating Adapters leveraging Wasserstein Barycenters for Finetuning Foundation Models in Multi-Modal Federated Learningcited by 6
  2. 2025MCAT: Visual Query-Based Localization of Standard Anatomical Clips in Fetal Ultrasound Videos Using Multi-Tier Class-Aware Token Transformercited by 2
  3. 2026OmniFM: Toward Modality-Robust and Task-Agnostic Federated Learning for Heterogeneous Medical Imagingcited by 1
  4. 2026Beyond Raw Context Transfer: Representation-based Federated Retrieval-Augmented Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025FedVLMBench: Benchmarking Federated Fine-Tuning of Vision-Language Models
  2. 2025FedVLM: Scalable Personalized Vision-Language Models through Federated Learning
  3. 2026Decoupled Training with Local Reinforcement Fine-Tuning in Federated Learning
  4. 2026UniFed-VLM: Federated Instruction Tuning for Vision-Language Models with Multiple Heterogeneity
  5. 2025TOFA: Training-Free One-Shot Federated Adaptation for Vision-Language Models
  6. 2025pFedMMA: Personalized Federated Fine-Tuning with Multi-Modal Adapter for Vision-Language Models
  7. 2025FOCoOp: Enhancing Out-of-Distribution Robustness in Federated Prompt Learning for Vision-Language Models
  8. 2026Replacing Parameters with Preferences: Federated Alignment of Heterogeneous Vision-Language Models