mlsanityData to 7 Oct 2026

Examining Modality Incongruity in Multimodal Federated Learning for Medical Vision and Language-based Disease Detection

arXiv 2402.05294 ↗First posted 7 Feb 2024cs.LG cs.AI cs.CLcited by 8

Not on the map. Why?

Abstract (arXiv)

Multimodal Federated Learning (MMFL) utilizes multiple modalities in each client to build a more powerful Federated Learning (FL) model than its unimodal counterpart. However, the impact of missing modality in different clients, also called modality incongruity, has been greatly overlooked. This paper, for the first time, analyses the impact of modality incongruity and reveals its connection with data heterogeneity across participating clients. We particularly inspect whether incongruent MMFL with unimodal and multimodal clients is more beneficial than unimodal FL. Furthermore, we examine three potential routes of addressing this issue. Firstly, we study the effectiveness of various self-attention mechanisms towards incongruity-agnostic information fusion in MMFL. Secondly, we introduce a modality imputation network (MIN) pre-trained in a multimodal client for modality translation in unimodal clients and investigate its potential towards mitigating the missing modality problem. Thirdly, we assess the capability of client-level and server-level regularization techniques towards mitigating modality incongruity effects. Experiments are conducted under several MMFL settings on two publicly available real-world datasets, MIMIC-CXR and Open-I, with Chest X-Ray and radiology reports.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 27 of its 55 references that are in the map. First posted on arXiv 7 Feb 2024.

Builds on

27 of 55 references.

References · 27

  1. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  2. 2020Rethinking Attention with Performerscited by 1,584
  3. 2021Model-Contrastive Federated Learningcited by 762
  4. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 678
  5. 2017Tensor Fusion Network for Multimodal Sentiment Analysiscited by 429
  6. 2022Multimodal Learning with Transformers: A Surveycited by 399
  7. 2019A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protectioncited by 371
  8. 2021Federated Learning on Non-IID Data: A Surveycited by 345
  9. 2018Efficient Low-rank Multimodal Fusion with Modality-Specific Factorscited by 268
  10. 2021SMIL: Multimodal Learning with Severely Missing Modalitycited by 243
  11. 2020FedBE: Making Bayesian Model Ensemble Applicable to Federated Learningcited by 176
  12. 2017Video Generation From Textcited by 153
15 more
  1. 2018Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalitiescited by 149
  2. 2021Multi-modal Understanding and Generation for Medical Images and Text via Vision-Language Pre-Trainingcited by 96
  3. 2023Multimodal Federated Learning via Contrastive Representation Ensemblecited by 85
  4. 2021Federated Learning: Opportunities and Challengescited by 83
  5. 2022FedDM: Iterative Distribution Matching for Communication-Efficient Federated Learningcited by 65
  6. 2021Collaborative Federated Learning For Healthcare: Multi-Modal COVID-19 Diagnosis at the Edgecited by 56
  7. 2020TransModality: An End2End Fusion Method with Transformer for Multimodal Sentiment Analysiscited by 40
  8. 2020Training Strategies to Handle Missing Modalities for Audio-Visual Expression Recognitioncited by 36
  9. 2021Multimodal Federated Learning on IoT Datacited by 32
  10. 2018Semi-supervised Deep Generative Modelling of Incomplete Multi-Modality Emotional Datacited by 29
  11. 2020On the Benefits of Early Fusion in Multimodal Representation Learningcited by 27
  12. 2023FedMEKT: Distillation-based Embedding Knowledge Transfer for Multimodal Federated Learningcited by 17
  13. 2019Multimodal and Multi-view Models for Emotion Recognitioncited by 10
  14. 2018Toward Marker-free 3D Pose Estimation in Lifting: A Deep Multi-view Solutioncited by 5
  15. 2023Rethinking Semi-Supervised Federated Learning: How to co-train fully-labeled and fully-unlabeled client imaging datacited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 3; 2026: 1.

8 papers.

  1. 2024Deep Multimodal Learning with Missing Modality: A Surveycited by 89
  2. 2025Multimodal Federated Learning With Missing Modalities through Feature Imputation Networkcited by 8
  3. 2024FedPIA -- Permuting and Integrating Adapters leveraging Wasserstein Barycenters for Finetuning Foundation Models in Multi-Modal Federated Learningcited by 6
  4. 2024Feasibility of Federated Learning from Client Databases with Different Brain Diseases and MRI Modalitiescited by 5
  5. 2024F3OCUS -- Federated Finetuning of Vision-Language Foundation Models with Optimal Client Layer Updating Strategy via Multi-objective Meta-Heuristicscited by 4
  6. 2025MCAT: Visual Query-Based Localization of Standard Anatomical Clips in Fetal Ultrasound Videos Using Multi-Tier Class-Aware Token Transformercited by 2
  7. 2025Robust Incomplete-Modality Alignment for Ophthalmic Disease Grading and Diagnosis via Labeled Optimal Transportcited by 0
  8. 2026ProMoE-FL: Prototype-conditioned Mixture of Experts for Multimodal Federated Learning with Missing Modalitiescited by 0