mlsanityData to 7 Oct 2026

Communication-Efficient Multimodal Federated Learning: Joint Modality and Client Selection

arXiv 2401.16685 ↗First posted 30 Jan 2024cs.LG cs.DCcited by 10

Not on the map. Why?

Abstract (arXiv)

Multimodal federated learning (MFL) aims to enrich model training in FL settings where clients are collecting measurements across multiple modalities. However, key challenges to MFL remain unaddressed, particularly in heterogeneous network settings where: (i) the set of modalities collected by each client is diverse, and (ii) communication limitations prevent clients from uploading all their locally trained modality encoders to the server. In this paper, we propose Multimodal Federated learning with joint Modality and Client selection (MFedMC), a communication-efficient MFL framework that tackles these challenges through a decoupled architecture and selective uploading. Unlike traditional holistic fusion approaches, MFedMC separates modality encoders and fusion modules: modality encoders are aggregated at the server for generalization across diverse client distributions, while fusion modules remain local to each client for personalized adaptation to individual modality configurations and data characteristics. Building on this decoupled design, our joint selection algorithm incorporates two main components: (a) A modality selection methodology for each client, which weighs (i) the impact of the modality, gauged by Shapley value analysis, (ii) the modality encoder size as a gauge of communication overhead, and (iii) the frequency of modality encoder updates, denoted recency, to enhance generalizability. (b) A client selection strategy for the server based on the local loss of modality encoders at each client. Experiments on five real-world datasets demonstrate that MFedMC achieves comparable accuracy to several baselines while reducing communication overhead by over 20×. A demo video and our code are available at https://liangqiy.com/mfedmc/.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 33 of its 75 references that are in the map. First posted on arXiv 30 Jan 2024.

Builds on

33 of 75 references.

References · 33

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  2. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  3. 2019Advances and Open Problems in Federated Learningcited by 3,021
  4. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
  5. 2018MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversationscited by 636
  6. 2019Robust and Communication-Efficient Federated Learning from Non-IID Datacited by 415
  7. 2019Federated Learning in Mobile Edge Networks: A Comprehensive Surveycited by 407
  8. 2019The many Shapley values for model explanationcited by 341
  9. 2020Client Selection in Federated Learning: Convergence Analysis and Power-of-Choice Selection Strategiescited by 196
  10. 2020Deep Learning for ECG Analysis: Benchmarks and Insights from PTB-XLcited by 158
  11. 2023Decentralized Federated Learning: A Survey and Perspectivecited by 86
  12. 2023Multimodal Federated Learning via Contrastive Representation Ensemblecited by 85
21 more
  1. 2021Tackling System and Statistical Heterogeneity for Federated Learning with Adaptive Client Samplingcited by 66
  2. 2022Client Selection in Federated Learning: Principles, Challenges, and Opportunitiescited by 62
  3. 2020FedHome: Cloud-Edge based Personalized Federated Learning for In-Home Health Monitoringcited by 61
  4. 2022Communication-Efficient Stochastic Zeroth-Order Optimization for Federated Learningcited by 59
  5. 2023FedMultimodal: A Benchmark For Multimodal Federated Learningcited by 51
  6. 2020TransModality: An End2End Fusion Method with Transformer for Multimodal Sentiment Analysiscited by 40
  7. 2023Federated Learning for Connected and Automated Vehicles: A Survey of Existing Approaches and Challengescited by 39
  8. 2021Multimodal Federated Learning on IoT Datacited by 32
  9. 2023AutoFed: Heterogeneity-Aware Federated Multimodal Learning for Robust Autonomous Drivingcited by 32
  10. 2020Energy Efficient Federated Learning over Heterogeneous Mobile Devices via Joint Design of Weight Quantization and Wireless Transmissioncited by 22
  11. 2024Cross-Modal Prototype based Multimodal Federated Learning under Severely Missing Modalitycited by 21
  12. 2023Peer-to-Peer Federated Continual Learning for Naturalistic Driving Action Recognitioncited by 19
  13. 2023A Survey of Federated Learning for Connected and Automated Vehiclescited by 17
  14. 2023Multimodal Federated Learning with Missing Modality via Prototype Mask and Contrastcited by 17
  15. 2023On the Convergence of Decentralized Federated Learning Under Imperfect Information Sharingcited by 16
  16. 2022Mitigating Biases in Student Performance Prediction via Attention-Based Personalized Federated Learningcited by 13
  17. 2023Federated Transfer-Ordered-Personalized Learning for Driver Monitoring Applicationcited by 11
  18. 2023FedMFS: Federated Multimodal Fusion Learning with Selective Modality Communicationcited by 4
  19. 2023Passive Radio Frequency-based 3D Indoor Positioning System via Ensemble Learningcited by 2
  20. 2023Digital Ethics in Federated Learningcited by 2
  21. 2023Device Sampling and Resource Optimization for Federated Learning in Cooperative Edge Networkscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 4; 2026: 3.

10 papers.

  1. 2024Non-IID data in Federated Learning: A Survey with Taxonomy, Metrics, Methods, Frameworks and Future Directionscited by 12
  2. 2024Prioritizing Modalities: Flexible Importance Scheduling in Federated Multimodal Learningcited by 4
  3. 2024Using Diffusion Models as Generative Replay in Continual Federated Learning -- What will Happen?cited by 2
  4. 2025FedGIN: Federated Learning with Dynamic Global Intensity Non-linear Augmentation for Organ Segmentation using Multi-modal Imagescited by 2
  5. 2025FedRecon: Missing Modality Reconstruction in Heterogeneous Distributed Environmentscited by 0
  6. 2025Multimodal Federated Learning: A Survey through the Lens of Different FL Paradigmscited by 0
  7. 2025HybridVFL: Disentangled Feature Learning for Edge-Enabled Vertical Federated Multimodal Classificationcited by 0
  8. 2026PID-Guided Partial Alignment for Multimodal Decentralized Federated Learningcited by 0
  9. 2026Optimizing Server Placement for Vertical Federated Learning in Dynamic Edge/Fog Networkscited by 0
  10. 2026Contribution-Aware Bandwidth Allocation for Multimodal Split Learningcited by 0