# Federated Pseudo Modality Generation for Incomplete Multi-Modal MRI Reconstruction

arXiv: https://arxiv.org/abs/2308.10910 · HTML: https://mlsanity.com/paper/2308.10910/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 62 references that are in the map.
- First posted on arXiv 20 Aug 2023.

## Builds on (most cited first, 30 of 33)

- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [On the Convergence of FedAvg on Non-IID Data](https://mlsanity.com/paper/1907.02189/) (2019)
- [Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization](https://mlsanity.com/paper/2007.07481/) (2020)
- [The Future of Digital Health with Federated Learning](https://mlsanity.com/paper/2003.08119/) (2020)
- [FedBN: Federated Learning on Non-IID Features via Local Batch Normalization](https://mlsanity.com/paper/2102.07623/) (2021)
- [FDA: Fourier Domain Adaptation for Semantic Segmentation](https://mlsanity.com/paper/2004.05498/) (2020)
- [fastMRI: An Open Dataset and Benchmarks for Accelerated MRI](https://mlsanity.com/paper/1811.08839/) (2018)
- [A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection](https://mlsanity.com/paper/1907.09693/) (2019)
- [Learning a Variational Network for Reconstruction of Accelerated MRI Data](https://mlsanity.com/paper/1704.00447/) (2017)
- [FedDG: Federated Domain Generalization on Medical Image Segmentation via Episodic Learning in Continuous Frequency Space](https://mlsanity.com/paper/2103.06030/) (2021)
- [TransBTS: Multimodal Brain Tumor Segmentation Using Transformer](https://mlsanity.com/paper/2103.04430/) (2021)
- [End-to-End Variational Networks for Accelerated MRI Reconstruction](https://mlsanity.com/paper/2004.06688/) (2020)
- [Privacy-preserving Federated Brain Tumour Segmentation](https://mlsanity.com/paper/1910.00962/) (2019)
- [HeMIS: Hetero-Modal Image Segmentation](https://mlsanity.com/paper/1607.05194/) (2016)
- [Missing MRI Pulse Sequence Synthesis using Multi-Modal Generative Adversarial Network](https://mlsanity.com/paper/1904.12200/) (2019)
- [Hi-Net: Hybrid-fusion Network for Multi-modal MR Image Synthesis](https://mlsanity.com/paper/2002.05000/) (2020)
- [Compressed Sensing MRI Reconstruction using a Generative Adversarial Network with a Cyclic Loss](https://mlsanity.com/paper/1709.00753/) (2017)
- [Deep Learning Methods for Parallel Magnetic Resonance Image Reconstruction](https://mlsanity.com/paper/1904.01112/) (2019)
- [Multi-institutional Collaborations for Improving Deep Learning-based Magnetic Resonance Image Reconstruction Using Federated Learning](https://mlsanity.com/paper/2103.02148/) (2021)
- [Unsupervised MRI Reconstruction via Zero-Shot Learned Adversarial Transformers](https://mlsanity.com/paper/2105.08059/) (2021)
- [Robust Multimodal Brain Tumor Segmentation via Feature Disentanglement and Gated Fusion](https://mlsanity.com/paper/2002.09708/) (2020)
- [HarmoFL: Harmonizing Local and Global Drifts in Federated Learning on Heterogeneous Medical Images](https://mlsanity.com/paper/2112.10775/) (2021)
- [Enhancing Pseudo Label Quality for Semi-Supervised Domain-Generalized Medical Image Segmentation](https://mlsanity.com/paper/2201.08657/) (2022)
- [Multi-Modal Transformer for Accelerated MR Imaging](https://mlsanity.com/paper/2106.14248/) (2021)
- [Federated Semi-supervised Medical Image Classification via Inter-client Relation Matching](https://mlsanity.com/paper/2106.08600/) (2021)
- [Specificity-Preserving Federated Learning for MR Image Reconstruction](https://mlsanity.com/paper/2112.05752/) (2021)
- [Federated Learning of Generative Image Priors for MRI Reconstruction](https://mlsanity.com/paper/2202.04175/) (2022)
- [Multimodal Federated Learning on IoT Data](https://mlsanity.com/paper/2109.04833/) (2021)
- [A Deep Information Sharing Network for Multi-contrast Compressed Sensing MRI Reconstruction](https://mlsanity.com/paper/1804.03596/) (2018)

## Built on by (most cited first, 5 of 5)

- [Multimodal Federated Learning in Healthcare: a Review](https://mlsanity.com/paper/2310.09650/) (2023)
- [CAR-MFL: Cross-Modal Augmentation by Retrieval for Multimodal Federated Learning with Missing Modalities](https://mlsanity.com/paper/2407.08648/) (2024)
- [Federated Learning for Large Models in Medical Imaging: A Comprehensive Review](https://mlsanity.com/paper/2508.20414/) (2025)
- [Purify-then-Align: Towards Robust Human Sensing under Modality Missing with Knowledge Distillation from Noisy Multimodal Teacher](https://mlsanity.com/paper/2604.05584/) (2026)
- [FedCoRe: Target-Adaptive Completion for Missing Modalities in Healthcare Federated Learning](https://mlsanity.com/paper/2608.18311/) (2026)

## Abstract (arXiv)

While multi-modal learning has been widely used for MRI reconstruction, it relies on paired multi-modal data which is difficult to acquire in real clinical scenarios. Especially in the federated setting, the common situation is that several medical institutions only have single-modal data, termed the modality missing issue. Therefore, it is infeasible to deploy a standard federated learning framework in such conditions. In this paper, we propose a novel communication-efficient federated learning framework, namely Fed-PMG, to address the missing modality challenge in federated multi-modal MRI reconstruction. Specifically, we utilize a pseudo modality generation mechanism to recover the missing modality for each single-modal client by sharing the distribution information of the amplitude spectrum in frequency space. However, the step of sharing the original amplitude spectrum leads to heavy communication costs. To reduce the communication cost, we introduce a clustering scheme to project the set of amplitude spectrum into finite cluster centroids, and share them among the clients. With such an elaborate design, our approach can effectively complete the missing modality within an acceptable communication cost. Extensive experiments demonstrate that our proposed method can attain similar performance with the ideal scenario, i.e., all clients have the full set of modalities. The source code will be released.