mlsanityData to 14 Oct 2026

FLoRA: Enhancing Vision-Language Models with Parameter-Efficient Federated Learning

arXiv 2404.15182 ↗First posted 12 Apr 2024cs.LG cs.AIcited by 16

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

In the rapidly evolving field of artificial intelligence, multimodal models, e.g., integrating vision and language into visual-language models (VLMs), have become pivotal for many applications, ranging from image captioning to multimodal search engines. Among these models, the Contrastive Language-Image Pre-training (CLIP) model has demonstrated remarkable performance in understanding and generating nuanced relationships between text and images. However, the conventional training of such models often requires centralized aggregation of vast datasets, posing significant privacy and data governance challenges. To address these concerns, this paper proposes a novel approach that leverages Federated Learning and parameter-efficient adapters, i.e., Low-Rank Adaptation (LoRA), to train VLMs. This methodology preserves data privacy by training models across decentralized data sources and ensures model adaptability and efficiency through LoRA's parameter-efficient fine-tuning. Our approach accelerates training time by up to 34.72 times and requires 2.47 times less memory usage than full fine-tuning.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 29 of its 43 references that are in the map. First posted on arXiv 12 Apr 2024.

Builds on

29 of 43 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Text-driven Prompt Generation for Vision-Language Models in Federated Learning9 Oct

Other references · 28

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  7. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  8. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  9. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  10. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  11. 2013Describing Textures in the Wildcited by 2,125
  12. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
16 more
  1. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  2. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
  3. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,423
  4. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,229
  5. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 906
  6. 2015Pointer Networkscited by 895
  7. 2021Robust fine-tuning of zero-shot modelscited by 797
  8. 2021Model-Contrastive Federated Learningcited by 766
  9. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 682
  10. 2021Federated Learning on Non-IID Data Silos: An Experimental Studycited by 568
  11. 2017A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasetscited by 377
  12. 2021Personalized Federated Learning using Hypernetworkscited by 212
  13. 2022Finetune like you pretrain: Improved finetuning of zero-shot vision modelscited by 150
  14. 2022PromptFL: Let Federated Participants Cooperatively Learn Prompts Instead of Models -- Federated Learning in Age of Foundation Modelcited by 134
  15. 2023FedCLIP: Fast Generalization and Personalization for CLIP in Federated Learningcited by 70
  16. 2023pFedLoRA: Model-Heterogeneous Personalized Federated Learning with LoRA Tuningcited by 40

Built on by

Cited by year

Citing papers per year: 2024: 3; 2025: 13.

16 papers.

Sort:

By area:

  1. 2025Vision-Language Models for Edge Networks: A Comprehensive Surveycited by 29
  2. 2024Federated LoRA with Sparse Communicationcited by 26
  3. 2025UAVs Meet Agentic AI: A Multidomain Survey of Autonomous Aerial Intelligence and Agentic UAVscited by 13
  4. 2025Distributed LLMs and Multimodal Large Language Models: A Survey on Advances, Challenges, and Future Directionscited by 10
  5. 2025VizGenie: Toward Self-Refining, Domain-Aware Workflows for Next-Generation Scientific Visualizationcited by 8
  6. 2024FedPIA -- Permuting and Integrating Adapters leveraging Wasserstein Barycenters for Finetuning Foundation Models in Multi-Modal Federated Learningcited by 6
  7. 2024DEeR: Deviation Eliminating and Noise Regulating for Privacy-preserving Federated Low-rank Adaptationcited by 4
  8. 2025Many-Task Federated Fine-Tuning via Unified Task Vectorscited by 3
  9. 2025VLLFL: A Vision-Language Model Based Lightweight Federated Learning Framework for Smart Agriculturecited by 3
  10. 2025FedVLM: Scalable Personalized Vision-Language Models through Federated Learningcited by 2
  11. 2025Federated Multimodal Learning with Dual Adapters and Selective Pruning for Communication and Computational Efficiencycited by 1
  12. 2025Foundational Models and Federated Learning: Survey, Taxonomy, Challenges and Practical Insightscited by 1
4 more
  1. 2025FHBench: Towards Efficient and Personalized Federated Learning for Multimodal Healthcarecited by 0
  2. 2025Frugal Federated Learning for Violence Detection: A Comparison of LoRA-Tuned VLMs and Personalized CNNscited by 0
  3. 2025FedReplay: A Feature Replay Assisted Federated Transfer Learning Framework for Efficient and Privacy-Preserving Smart Agriculturecited by 0
  4. 2025Federated Learning for Video Violence Detection: Complementary Roles of Lightweight CNNs and Vision-Language Models for Energy-Efficient Usecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025TOFA: Training-Free One-Shot Federated Adaptation for Vision-Language Models
  2. 2025FedVLMBench: Benchmarking Federated Fine-Tuning of Vision-Language Models
  3. 2026Rethinking LoRA for Privacy-Preserving Federated Learning in Large Models
  4. 2025LangVision-LoRA-NAS: Neural Architecture Search for Variable LoRA Rank in Vision Language Models
  5. 2026BLEND: Balancing Personalization vs. Generalization in Federated Vision-Language Models
  6. 2025An Empirical Study of Federated Prompt Learning for Vision Language Model
  7. 2026UniFed-VLM: Federated Instruction Tuning for Vision-Language Models with Multiple Heterogeneity
  8. 2024MLLM-LLaVA-FL: Multimodal Large Language Model Assisted Federated Learning