# FedMM: Federated Multi-Modal Learning with Modality Heterogeneity in Computational Pathology

arXiv: https://arxiv.org/abs/2402.15858 · HTML: https://mlsanity.com/paper/2402.15858/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 36 references that are in the map.
- First posted on arXiv 24 Feb 2024.

## Builds on (most cited first, 18 of 18)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Attention-based Deep Multiple Instance Learning](https://mlsanity.com/paper/1802.04712/) (2018)
- [Self-Normalizing Neural Networks](https://mlsanity.com/paper/1706.02515/) (2017)
- [FedBN: Federated Learning on Non-IID Features via Local Batch Normalization](https://mlsanity.com/paper/2102.07623/) (2021)
- [Federated Learning on Non-IID Data Silos: An Experimental Study](https://mlsanity.com/paper/2102.02079/) (2021)
- [FedProto: Federated Prototype Learning across Heterogeneous Clients](https://mlsanity.com/paper/2105.00243/) (2021)
- [Federated Learning for Smart Healthcare: A Survey](https://mlsanity.com/paper/2111.08834/) (2021)
- [Pathomic Fusion: An Integrated Framework for Fusing Histopathology and Genomic Features for Cancer Diagnosis and Prognosis](https://mlsanity.com/paper/1912.08937/) (2019)
- [Multi-site fMRI Analysis Using Privacy-preserving Federated Learning and Domain Adaptation: ABIDE Results](https://mlsanity.com/paper/2001.05647/) (2020)
- [Rethinking Architecture Design for Tackling Data Heterogeneity in Federated Learning](https://mlsanity.com/paper/2106.06047/) (2021)
- [Multimodal Federated Learning via Contrastive Representation Ensemble](https://mlsanity.com/paper/2302.08888/) (2023)
- [Decentralized Federated Learning through Proxy Model Sharing](https://mlsanity.com/paper/2111.11343/) (2021)
- [Federated Learning for Computational Pathology on Gigapixel Whole Slide Images](https://mlsanity.com/paper/2009.10190/) (2020)
- [A Comparison of Five Multiple Instance Learning Pooling Functions for Sound Event Detection with Weak Labeling](https://mlsanity.com/paper/1810.09050/) (2018)
- [Rethinking Data Heterogeneity in Federated Learning: Introducing a New Notion and Standard Benchmarks](https://mlsanity.com/paper/2209.15595/) (2022)
- [CAD: Co-Adapting Discriminative Features for Improved Few-Shot Classification](https://mlsanity.com/paper/2203.13465/) (2022)

## Built on by (most cited first, 17 of 17)

- [Deep Multimodal Learning with Missing Modality: A Survey](https://mlsanity.com/paper/2409.07825/) (2024)
- [LoRA-FAIR: Federated LoRA Fine-Tuning with Aggregation and Initialization Refinement](https://mlsanity.com/paper/2411.14961/) (2024)
- [FedMAC: Tackling Partial-Modality Missing in Federated Learning with Cross-Modal Aggregation and Contrastive Regularization](https://mlsanity.com/paper/2410.03070/) (2024)
- [Open Challenges and Opportunities in Federated Foundation Models Towards Biomedical Healthcare](https://mlsanity.com/paper/2405.06784/) (2024)
- [MM-OpenFGL: A Comprehensive Benchmark for Multimodal Federated Graph Learning](https://mlsanity.com/paper/2601.22416/) (2026)
- [Learning Reconfigurable Representations for Multimodal Federated Learning with Missing Data](https://mlsanity.com/paper/2510.22880/) (2025)
- [FedTreeLoRA: Reconciling Statistical and Functional Heterogeneity in Federated LoRA Fine-Tuning](https://mlsanity.com/paper/2603.13282/) (2026)
- [FSSUAVL: A Discriminative Framework using Vision Models for Federated Self-Supervised Audio and Image Understanding](https://mlsanity.com/paper/2504.09516/) (2025)
- [Towards Robust Federated Multimodal Graph Learning under Modality Heterogeneity](https://mlsanity.com/paper/2605.12584/) (2026)
- [Adaptive Prototype Knowledge Transfer for Federated Learning with Mixed Modalities and Heterogeneous Tasks](https://mlsanity.com/paper/2502.04400/) (2025)
- [Multimodal Federated Learning: A Survey through the Lens of Different FL Paradigms](https://mlsanity.com/paper/2505.21792/) (2025)
- [Role-Aware Multi-modal federated learning system for detecting phishing webpages](https://mlsanity.com/paper/2509.22369/) (2025)
- [Latency-aware Multimodal Federated Learning over UAV Networks](https://mlsanity.com/paper/2510.01717/) (2025)
- [Conditional Imputation for Within-Modality Missingness in Multi-Modal Federated Learning](https://mlsanity.com/paper/2604.23112/) (2026)
- [Multimodal Graph Negative Learning](https://mlsanity.com/paper/2606.12863/) (2026)
- [Towards Modality-imbalanced Federated Graph Learning: A Data Synthesis-based Approach](https://mlsanity.com/paper/2606.20382/) (2026)
- [MOSAIC: Modality-agnostic Spectral Alignment for Federated Image-level Weakly Supervised Tumor Segmentation under Client-specific Missing Modalities](https://mlsanity.com/paper/2608.19788/) (2026)

## Abstract (arXiv)

The fusion of complementary multimodal information is crucial in computational pathology for accurate diagnostics. However, existing multimodal learning approaches necessitate access to users' raw data, posing substantial privacy risks. While Federated Learning (FL) serves as a privacy-preserving alternative, it falls short in addressing the challenges posed by heterogeneous (yet possibly overlapped) modalities data across various hospitals. To bridge this gap, we propose a Federated Multi-Modal (FedMM) learning framework that federatedly trains multiple single-modal feature extractors to enhance subsequent classification performance instead of existing FL that aims to train a unified multimodal fusion model. Any participating hospital, even with small-scale datasets or limited devices, can leverage these federated trained extractors to perform local downstream tasks (e.g., classification) while ensuring data privacy. Through comprehensive evaluations of two publicly available datasets, we demonstrate that FedMM notably outperforms two baselines in accuracy and AUC metrics.