mlsanityData to 7 Oct 2026

FedMLLM: Federated Fine-tuning MLLM on Multimodal Heterogeneity Data

arXiv 2411.14717 ↗First posted 22 Nov 2024cs.LG cs.CL cs.CVcited by 7

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) have made significant advancements, demonstrating powerful capabilities in processing and understanding multimodal data. Fine-tuning MLLMs with Federated Learning (FL) allows for expanding the training data scope by including private data sources, thereby enhancing their practical applicability in privacy-sensitive domains. However, current research remains in the early stage, particularly in addressing the multimodal heterogeneities in real-world applications. In this paper, we introduce a benchmark to evaluate the performance of federated fine-tuning of MLLMs across various multimodal heterogeneous scenarios, laying the groundwork for future research in the field. Our benchmark includes two lightweight MLLMs, two downstream tasks, three evaluation metrics, and five datasets across three domains, along with six comparison baselines, covering over ten types of modality heterogeneities across four multimodal scenarios. To address the challenges posed by multimodal heterogeneity, we develop a general FedMLLM framework that integrates classic FL methods alongside two modality-agnostic strategies. Extensive experimental results show that our proposed FL paradigm improves the performance of MLLMs by broadening the range of training data and mitigating multimodal heterogeneity. Code is available in supplementary materials.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 35 of its 53 references that are in the map. First posted on arXiv 22 Nov 2024.

Builds on

35 of 53 references.

References · 35

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  7. 2019Advances and Open Problems in Federated Learningcited by 3,021
  8. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  10. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  11. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  12. 2020Adaptive Federated Optimizationcited by 1,036
23 more
  1. 2023A Survey on Multimodal Large Language Modelscited by 898
  2. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 836
  3. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 485
  4. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 444
  5. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 361
  6. 2023MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Datacited by 282
  7. 2021SMIL: Multimodal Learning with Severely Missing Modalitycited by 243
  8. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 216
  9. 2023Multimodal Large Language Models: A Surveycited by 193
  10. 2023Towards Building the Federated GPT: Federated Instruction Tuningcited by 184
  11. 2023FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learningcited by 146
  12. 2024OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learningcited by 124
  13. 2023Federated Learning for Generalization, Robustness, Fairness: A Survey and Benchmarkcited by 97
  14. 2024Efficient Multimodal Large Language Models: A Surveycited by 95
  15. 2018CrisisMMD: Multimodal Twitter Datasets from Natural Disasterscited by 88
  16. 2023Multimodal Federated Learning via Contrastive Representation Ensemblecited by 85
  17. 2021Federated Learning: Opportunities and Challengescited by 83
  18. 2023FedDAT: An Approach for Foundation Model Finetuning in Multi-Modal Heterogeneous Federated Learningcited by 59
  19. 2023FedMultimodal: A Benchmark For Multimodal Federated Learningcited by 51
  20. 2022Cross-Silo Federated Learning: Challenges and Opportunitiescited by 38
  21. 2023L-DAWA: Layer-wise Divergence Aware Weight Aggregation in Federated Self-Supervised Visual Representation Learningcited by 29
  22. 2024PeFoMed: Parameter Efficient Fine-tuning of Multimodal Large Language Models for Medical Imagingcited by 21
  23. 2024Leveraging Foundation Models for Multi-modal Federated Learning with Incomplete Modalitycited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 4.

7 papers.

  1. 2025FedVLMBench: Benchmarking Federated Fine-Tuning of Vision-Language Modelscited by 6
  2. 2025Multimodal Representation Learning and Fusioncited by 3
  3. 2025FedNano: Toward Lightweight Federated Tuning for Pretrained Multimodal Large Language Modelscited by 1
  4. 2026A Step Toward Federated Pretraining of Multimodal Large Language Modelscited by 1
  5. 2026FedUMM: A General Framework for Federated Learning with Unified Multimodal Modelscited by 0
  6. 2026PFAdapter: Hierarchical LoRA Decomposition for Personalized Federated MLLMscited by 0
  7. 2026Focus Where It Counts: A Salience-Driven Vision-Language Model for Low Vision Assistancecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024MLLM-LLaVA-FL: Multimodal Large Language Model Assisted Federated Learning
  2. 2024Personalized Multimodal Large Language Models: A Survey
  3. 2025OrchMLLM: Orchestrate Multimodal Data with Batch Post-Balancing to Accelerate Multimodal Large Language Model Training
  4. 2026Federated Prompt Learning: A Unified Framework, Empirical Analysis, and Future Directions
  5. 2024An Empirical Study on Parameter-Efficient Fine-Tuning for MultiModal Large Language Models
  6. 2025FedALT: Federated Fine-Tuning through Adaptive Local Training with Rest-of-World LoRA
  7. 2024FLoRA: Enhancing Vision-Language Models with Parameter-Efficient Federated Learning
  8. 2026Open-MMUnlearning: Unifying Methods and Evaluation for MLLM Unlearning