# CAR-MFL: Cross-Modal Augmentation by Retrieval for Multimodal Federated Learning with Missing Modalities

arXiv: https://arxiv.org/abs/2407.08648 · HTML: https://mlsanity.com/paper/2407.08648/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 32 references that are in the map.
- First posted on arXiv 11 Jul 2024.

## Builds on (most cited first, 20 of 20)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [Multi-modal Understanding and Generation for Medical Images and Text via Vision-Language Pre-Training](https://mlsanity.com/paper/2105.11333/) (2021)
- [Multimodal Federated Learning via Contrastive Representation Ensemble](https://mlsanity.com/paper/2302.08888/) (2023)
- [CXR-CLIP: Toward Large Scale Chest X-ray Language-Image Pre-training](https://mlsanity.com/paper/2310.13292/) (2023)
- [Towards Fair Federated Learning with Zero-Shot Data Augmentation](https://mlsanity.com/paper/2104.13417/) (2021)
- [Collaborative Federated Learning For Healthcare: Multi-Modal COVID-19 Diagnosis at the Edge](https://mlsanity.com/paper/2101.07511/) (2021)
- [Generative Text-Guided 3D Vision-Language Pretraining for Unified Medical Image Segmentation](https://mlsanity.com/paper/2306.04811/) (2023)
- [Towards Unifying Medical Vision-and-Language Pre-training via Soft Prompts](https://mlsanity.com/paper/2302.08958/) (2023)
- [AutoFed: Heterogeneity-Aware Federated Multimodal Learning for Robust Autonomous Driving](https://mlsanity.com/paper/2302.08646/) (2023)
- [Breaking with Fixed Set Pathology Recognition through Report-Guided Contrastive Training](https://mlsanity.com/paper/2205.07139/) (2022)
- [Medical Vision Language Pretraining: A survey](https://mlsanity.com/paper/2312.06224/) (2023)
- [Cross-Modal Prototype based Multimodal Federated Learning under Severely Missing Modality](https://mlsanity.com/paper/2401.13898/) (2024)
- [A unified representation network for segmentation with missing modalities](https://mlsanity.com/paper/1908.06683/) (2019)
- [Multimodal Federated Learning in Healthcare: a Review](https://mlsanity.com/paper/2310.09650/) (2023)
- [Federated Uncertainty-Aware Aggregation for Fundus Diabetic Retinopathy Staging](https://mlsanity.com/paper/2303.13033/) (2023)
- [Federated Pseudo Modality Generation for Incomplete Multi-Modal MRI Reconstruction](https://mlsanity.com/paper/2308.10910/) (2023)

## Built on by (most cited first, 5 of 5)

- [Multimodal Federated Learning With Missing Modalities through Feature Imputation Network](https://mlsanity.com/paper/2505.20232/) (2025)
- [Med-MMFL: A Multimodal Federated Learning Benchmark in Healthcare](https://mlsanity.com/paper/2602.04416/) (2026)
- [Adaptive Prototype Knowledge Transfer for Federated Learning with Mixed Modalities and Heterogeneous Tasks](https://mlsanity.com/paper/2502.04400/) (2025)
- [ProMoE-FL: Prototype-conditioned Mixture of Experts for Multimodal Federated Learning with Missing Modalities](https://mlsanity.com/paper/2607.06633/) (2026)
- [FedCoRe: Target-Adaptive Completion for Missing Modalities in Healthcare Federated Learning](https://mlsanity.com/paper/2608.18311/) (2026)

## Abstract (arXiv)

Multimodal AI has demonstrated superior performance over unimodal approaches by leveraging diverse data sources for more comprehensive analysis. However, applying this effectiveness in healthcare is challenging due to the limited availability of public datasets. Federated learning presents an exciting solution, allowing the use of extensive databases from hospitals and health centers without centralizing sensitive data, thus maintaining privacy and security. Yet, research in multimodal federated learning, particularly in scenarios with missing modalities a common issue in healthcare datasets remains scarce, highlighting a critical area for future exploration. Toward this, we propose a novel method for multimodal federated learning with missing modalities. Our contribution lies in a novel cross-modal data augmentation by retrieval, leveraging the small publicly available dataset to fill the missing modalities in the clients. Our method learns the parameters in a federated manner, ensuring privacy protection and improving performance in multiple challenging multimodal benchmarks in the medical domain, surpassing several competitive baselines. Code Available: https://github.com/bhattarailab/CAR-MFL