# Cross-Modal Vertical Federated Learning for MRI Reconstruction

arXiv: https://arxiv.org/abs/2306.02673 · HTML: https://mlsanity.com/paper/2306.02673/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 36 references that are in the map.
- First posted on arXiv 5 Jun 2023.

## Builds on (most cited first, 22 of 22)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Federated Learning: Challenges, Methods, and Future Directions](https://mlsanity.com/paper/1908.07873/) (2019)
- [On the Convergence of FedAvg on Non-IID Data](https://mlsanity.com/paper/1907.02189/) (2019)
- [Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challenge](https://mlsanity.com/paper/1811.02629/) (2018)
- [fastMRI: An Open Dataset and Benchmarks for Accelerated MRI](https://mlsanity.com/paper/1811.08839/) (2018)
- [End-to-End Variational Networks for Accelerated MRI Reconstruction](https://mlsanity.com/paper/2004.06688/) (2020)
- [SecureBoost: A Lossless Federated Learning Framework](https://mlsanity.com/paper/1901.08755/) (2019)
- [Domain Agnostic Learning with Disentangled Representations](https://mlsanity.com/paper/1904.12347/) (2019)
- [Multi-institutional Collaborations for Improving Deep Learning-based Magnetic Resonance Image Reconstruction Using Federated Learning](https://mlsanity.com/paper/2103.02148/) (2021)
- [Unsupervised MRI Reconstruction via Zero-Shot Learned Adversarial Transformers](https://mlsanity.com/paper/2105.08059/) (2021)
- [Robust Multimodal Brain Tumor Segmentation via Feature Disentanglement and Gated Fusion](https://mlsanity.com/paper/2002.09708/) (2020)
- [Task Transformer Network for Joint MRI Reconstruction and Super-Resolution](https://mlsanity.com/paper/2106.06742/) (2021)
- [Unsupervised Deformable Registration for Multi-Modal Images via Disentangled Representations](https://mlsanity.com/paper/1903.09331/) (2019)
- [GrappaNet: Combining Parallel Imaging with Deep Learning for Multi-Coil MRI Reconstruction](https://mlsanity.com/paper/1910.12325/) (2019)
- [Specificity-Preserving Federated Learning for MR Image Reconstruction](https://mlsanity.com/paper/2112.05752/) (2021)
- [Federated Contrastive Learning for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/2204.10983/) (2022)
- [Unsupervised MRI Reconstruction with Generative Adversarial Networks](https://mlsanity.com/paper/2008.13065/) (2020)
- [Entity Resolution and Federated Learning get a Federated Resolution](https://mlsanity.com/paper/1803.04035/) (2018)
- [Representation Disentanglement for Multi-modal brain MR Analysis](https://mlsanity.com/paper/2102.11456/) (2021)
- [Factorised spatial representation learning: application in semi-supervised myocardial segmentation](https://mlsanity.com/paper/1803.07031/) (2018)
- [Deep Learning based Multi-modal Computing with Feature Disentanglement for MRI Image Synthesis](https://mlsanity.com/paper/2105.02835/) (2021)

## Built on by (most cited first, 9 of 9)

- [SAM-MPA: Applying SAM to Few-shot Medical Image Segmentation using Mask Propagation and Auto-prompting](https://mlsanity.com/paper/2411.17363/) (2024)
- [Zero-shot capability of SAM-family models for bone segmentation in CT scans](https://mlsanity.com/paper/2411.08629/) (2024)
- [ZALM3: Zero-Shot Enhancement of Vision-Language Alignment via In-Context Information in Multi-Turn Multimodal Medical Dialogue](https://mlsanity.com/paper/2409.17610/) (2024)
- [Assessing Foundational Medical 'Segment Anything' (Med-SAM1, Med-SAM2) Deep Learning Models for Left Atrial Segmentation in 3D LGE MRI](https://mlsanity.com/paper/2411.05963/) (2024)
- [A SAM-guided and Match-based Semi-Supervised Segmentation Framework for Medical Imaging](https://mlsanity.com/paper/2411.16949/) (2024)
- [Exploiting the Segment Anything Model (SAM) for Lung Segmentation in Chest X-ray Images](https://mlsanity.com/paper/2411.03064/) (2024)
- [Behavior Backdoor for Deep Learning Models](https://mlsanity.com/paper/2412.01369/) (2024)
- [FedRecon: Missing Modality Reconstruction in Heterogeneous Distributed Environments](https://mlsanity.com/paper/2504.09941/) (2025)
- [HybridVFL: Disentangled Feature Learning for Edge-Enabled Vertical Federated Multimodal Classification](https://mlsanity.com/paper/2512.10701/) (2025)

## Abstract (arXiv)

Federated learning enables multiple hospitals to cooperatively learn a shared model without privacy disclosure. Existing methods often take a common assumption that the data from different hospitals have the same modalities. However, such a setting is difficult to fully satisfy in practical applications, since the imaging guidelines may be different between hospitals, which makes the number of individuals with the same set of modalities limited. To this end, we formulate this practical-yet-challenging cross-modal vertical federated learning task, in which shape data from multiple hospitals have different modalities with a small amount of multi-modality data collected from the same individuals. To tackle such a situation, we develop a novel framework, namely Federated Consistent Regularization constrained Feature Disentanglement (Fed-CRFD), for boosting MRI reconstruction by effectively exploring the overlapping samples (individuals with multi-modalities) and solving the domain shift problem caused by different modalities. Particularly, our Fed-CRFD involves an intra-client feature disentangle scheme to decouple data into modality-invariant and modality-specific features, where the modality-invariant features are leveraged to mitigate the domain shift problem. In addition, a cross-client latent representation consistency constraint is proposed specifically for the overlapping samples to further align the modality-invariant features extracted from different modalities. Hence, our method can fully exploit the multi-source data from hospitals while alleviating the domain shift problem. Extensive experiments on two typical MRI datasets demonstrate that our network clearly outperforms state-of-the-art MRI reconstruction methods. The source code will be publicly released upon the publication of this work.