Multimodal Federated Learning: A Survey through the Lens of Different FL Paradigms
Not on the map. Why?
Abstract (arXiv)
Multimodal Federated Learning (MFL) lies at the intersection of two pivotal research areas: leveraging complementary information from multiple modalities to improve downstream inference performance and enabling distributed training to enhance efficiency and preserve privacy. Despite the growing interest in MFL, there is currently no comprehensive taxonomy that organizes MFL through the lens of different Federated Learning (FL) paradigms. This perspective is important because multimodal data introduces distinct challenges across various FL settings. These challenges, including modality heterogeneity, privacy heterogeneity, and communication inefficiency, are fundamentally different from those encountered in traditional unimodal or non-FL scenarios. In this paper, we systematically examine MFL within the context of three major FL paradigms: horizontal FL (HFL), vertical FL (VFL), and hybrid FL. For each paradigm, we present the problem formulation, review representative training algorithms, and highlight the most prominent challenge introduced by multimodal data in distributed settings. We also discuss open challenges and provide insights for future research. By establishing this taxonomy, we aim to uncover the novel challenges posed by multimodal data from the perspective of different FL paradigms and to offer a new lens through which to understand and advance the development of MFL.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 80 of its 123 references that are in the map. First posted on arXiv 27 May 2025.
Builds on
80 of 123 references.
References · 80
- 2017Attention Is All You Needcited by 60,361
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2014Microsoft COCO: Common Objects in Contextcited by 17,608
- 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
- 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
- 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
- 2019Advances and Open Problems in Federated Learningcited by 3,046
- 2018Federated Optimization in Heterogeneous Networkscited by 2,987
- 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
- 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
- 2017The Kinetics Human Action Video Datasetcited by 1,909
38 more
- 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
- 2019Contrastive Multiview Codingcited by 1,301
- 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
- 2020Adaptive Federated Optimizationcited by 1,046
- 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 906
- 2018How To Backdoor Federated Learningcited by 765
- 2017Federated Multi-Task Learningcited by 672
- 2018MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversationscited by 643
- 2021FedBN: Federated Learning on Non-IID Features via Local Batch Normalizationcited by 571
- 2019Federated Learning with Personalization Layerscited by 541
- 2021Exploiting Shared Representations for Personalized Federated Learningcited by 541
- 2019Federated Learning with Differential Privacy: Algorithms and Performance Analysiscited by 536
- 2018Exploiting Unintended Feature Leakage in Collaborative Learningcited by 520
- 2021Towards Personalized Federated Learningcited by 464
- 2017Differentially Private Federated Learning: A Client Level Perspectivecited by 459
- 2019FedMD: Heterogenous Federated Learning via Model Distillationcited by 459
- 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 425
- 2018Split learning for health: Distributed deep learning without sharing raw patient datacited by 395
- 2019FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantizationcited by 356
- 2018Client Selection for Federated Learning with Heterogeneous Resources in Mobile Edgecited by 352
- 2017Deep Models Under the GAN: Information Leakage from Collaborative Deep Learningcited by 334
- 2021See through Gradients: Image Batch Recovery via GradInversioncited by 326
- 2020SplitFed: When Federated Learning Meets Split Learningcited by 310
- 2019Asynchronous Federated Optimizationcited by 267
- 2017Sparse Communication for Distributed Gradient Descentcited by 249
- 2018Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Datacited by 243
- 2019Client-Edge-Cloud Hierarchical Federated Learningcited by 202
- 2021ResViT: Residual vision transformers for multi-modal medical image synthesiscited by 177
- 2021Embodied Intelligence via Learning and Evolutioncited by 163
- 2020LDP-Fed: Federated Learning with Local Differential Privacycited by 132
- 2020Federated Semi-Supervised Learning with Inter-Client Consistency & Disjoint Learningcited by 112
- 2022Vertical Federated Learning: Concepts, Advances and Challengescited by 108
- 2020Feature Inference Attack on Model Predictions in Vertical Federated Learningcited by 106
- 2021Label Leakage and Protection in Two-party Split Learningcited by 90
- 2023Multimodal Federated Learning via Contrastive Representation Ensemblecited by 86
- 2021ML-Doctor: Holistic Risk Assessment of Inference Attacks Against Machine Learning Modelscited by 85
- 2019Overlearning Reveals Sensitive Attributescited by 76
- 2021CAFE: Catastrophic Data Leakage in Vertical Federated Learningcited by 76
50 most cited of 80 All 80 (JSON)
Built on by
Not cited yet.