mlsanityData to 7 Oct 2026

Towards Multi-modal Transformers in Federated Learning

arXiv 2404.12467 ↗First posted 18 Apr 2024cs.CV cs.LGcited by 12

Not on the map. Why?

Abstract (arXiv)

Multi-modal transformers mark significant progress in different domains, but siloed high-quality data hinders their further improvement. To remedy this, federated learning (FL) has emerged as a promising privacy-preserving paradigm for training models without direct access to the raw data held by different clients. Despite its potential, a considerable research direction regarding the unpaired uni-modal clients and the transformer architecture in FL remains unexplored. To fill this gap, this paper explores a transfer multi-modal federated learning (MFL) scenario within the vision-language domain, where clients possess data of various modalities distributed across different datasets. We systematically evaluate the performance of existing methods when a transformer architecture is utilized and introduce a novel framework called Federated modality complementary and collaboration (FedCola) by addressing the in-modality and cross-modality gaps among clients. Through extensive experiments across various FL settings, FedCola demonstrates superior performance over previous approaches, offering new perspectives on future federated training of multi-modal transformers.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 50 of its 69 references that are in the map. First posted on arXiv 18 Apr 2024.

Builds on

50 of 69 references.

References · 50

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  6. 2019Advances and Open Problems in Federated Learningcited by 3,021
  7. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  8. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  9. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  10. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  11. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 836
  12. 2021Model-Contrastive Federated Learningcited by 762
38 more
  1. 2021MedMNIST v2 -- A large-scale lightweight benchmark for 2D and 3D biomedical image classificationcited by 682
  2. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 678
  3. 2020Ditto: Fair and Robust Federated Learning Through Personalizationcited by 665
  4. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  5. 2021FedBN: Federated Learning on Non-IID Features via Local Batch Normalizationcited by 565
  6. 2020Federated Learning with Matched Averagingcited by 564
  7. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  8. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  9. 2021Towards Personalized Federated Learningcited by 462
  10. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424
  11. 2019Fair Resource Allocation in Federated Learningcited by 389
  12. 2020Adaptive Personalized Federated Learningcited by 353
  13. 2019Understanding and Robustifying Differentiable Architecture Searchcited by 176
  14. 2022Federated Learning from Pre-Trained Models: A Contrastive Learning Approachcited by 141
  15. 2021Local Learning Matters: Rethinking Data Heterogeneity in Federated Learningcited by 114
  16. 2021Rethinking Architecture Design for Tackling Data Heterogeneity in Federated Learningcited by 111
  17. 2020Stabilizing Differentiable Architecture Search via Perturbation-based Regularizationcited by 102
  18. 2023Multimodal Federated Learning via Contrastive Representation Ensemblecited by 85
  19. 2023When Foundation Model Meets Federated Learning: Motivations, Challenges, and Future Directionscited by 83
  20. 2022Heterogeneous Ensemble Knowledge Transfer for Training Large Models in Federated Learningcited by 81
  21. 2022On the Importance and Applicability of Pre-Training for Federated Learningcited by 71
  22. 2022Where to Begin? On the Impact of Pre-Training and Initialization in Federated Learningcited by 71
  23. 2023FedMultimodal: A Benchmark For Multimodal Federated Learningcited by 51
  24. 2021FedGEMS: Federated Learning of Larger Server Models via Selective Knowledge Fusioncited by 49
  25. 2022Evaluating Unsupervised Text Classification: Zero-shot and Similarity-based Approachescited by 39
  26. 2022FedTP: Federated Learning by Transformer Personalizationcited by 36
  27. 2021Multimodal Federated Learning on IoT Datacited by 32
  28. 2019On the Discrepancy between the Theoretical Analysis and Practical Implementations of Compressed Communication for Distributed Deep Learningcited by 31
  29. 2023FedPerfix: Towards Partial Model Personalization of Vision Transformers in Federated Learningcited by 21
  30. 2023Multimodal Federated Learning with Missing Modality via Prototype Mask and Contrastcited by 17
  31. 2024FedMef: Towards Memory-efficient Federated Dynamic Pruningcited by 14
  32. 2022PGFed: Personalize Each Client's Global Objective for Federated Learningcited by 13
  33. 2021Connecting Low-Loss Subspace for Personalized Federated Learningcited by 12
  34. 2022Exploring Parameter-Efficient Fine-Tuning to Enable Foundation Models in Federated Learningcited by 11
  35. 2023FedWon: Triumphing Multi-domain Federated Learning Without Normalizationcited by 11
  36. 2022FedGradNorm: Personalized Federated Gradient-Normalized Multi-Task Learningcited by 9
  37. 2024Multimodal Pathway: Improve Transformers with Irrelevant Data from Other Modalitiescited by 7
  38. 2022Personalized Federated Learning with Communication Compressioncited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 8; 2026: 3.

12 papers.

  1. 2025Vision-Language Models for Edge Networks: A Comprehensive Surveycited by 29
  2. 2025Multimodal Federated Learning With Missing Modalities through Feature Imputation Networkcited by 8
  3. 2025FedMVP: Federated Multimodal Visual Prompt Tuning for Vision-Language Modelscited by 4
  4. 2024Fibottention: Inceptive Visual Representation Learning with Diverse Attention Across Headscited by 3
  5. 2025FSSUAVL: A Discriminative Framework using Vision Models for Federated Self-Supervised Audio and Image Understandingcited by 1
  6. 2025Adaptive Prototype Knowledge Transfer for Federated Learning with Mixed Modalities and Heterogeneous Taskscited by 0
  7. 2025Closer to Reality: Practical Semi-Supervised Federated Learning for Foundation Model Adaptationcited by 0
  8. 2025FedMMKT:Co-Enhancing a Server Text-to-Image Model and Client Task Models in Multi-Modal Federated Learningcited by 0
  9. 2025How (Mis)calibrated is your Federated CLIP and what to do about it?cited by 0
  10. 2026Federated learning for unpaired multimodal data through a homogeneous transformer modelcited by 0
  11. 2026ProMoE-FL: Prototype-conditioned Mixture of Experts for Multimodal Federated Learning with Missing Modalitiescited by 0
  12. 2026Entropy-Guided Tensor Compression for Multimodal Federated Learning on Edge Devicescited by 0