mlsanityData to 14 Oct 2026

Federated Modality-specific Encoders and Multimodal Anchors for Personalized Brain Tumor Segmentation

arXiv 2403.11803 ↗First posted 18 Mar 2024cs.CVcited by 8

Not on the map. Why?

Abstract (arXiv)

Most existing federated learning (FL) methods for medical image analysis only considered intramodal heterogeneity, limiting their applicability to multimodal imaging applications. In practice, it is not uncommon that some FL participants only possess a subset of the complete imaging modalities, posing inter-modal heterogeneity as a challenge to effectively training a global model on all participants' data. In addition, each participant would expect to obtain a personalized model tailored for its local data characteristics from the FL in such a scenario. In this work, we propose a new FL framework with federated modality-specific encoders and multimodal anchors (FedMEMA) to simultaneously address the two concurrent issues. Above all, FedMEMA employs an exclusive encoder for each modality to account for the inter-modal heterogeneity in the first place. In the meantime, while the encoders are shared by the participants, the decoders are personalized to meet individual needs. Specifically, a server with full-modal data employs a fusion decoder to aggregate and fuse representations from all modality-specific encoders, thus bridging the modalities to optimize the encoders via backpropagation reversely. Meanwhile, multiple anchors are extracted from the fused multimodal representations and distributed to the clients in addition to the encoder parameters. On the other end, the clients with incomplete modalities calibrate their missing-modal representations toward the global full-modal anchors via scaled dot-product cross-attention, making up the information loss due to absent modalities while adapting the representations of present ones. FedMEMA is validated on the BraTS 2020 benchmark for multimodal brain tumor segmentation. Results show that it outperforms various up-to-date methods for multimodal and personalized FL and that its novel designs are effective. Our code is available.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 22 of its 37 references that are in the map. First posted on arXiv 18 Mar 2024.

Builds on

22 of 37 references.

References · 22

  1. 2017Attention Is All You Needcited by 60,361
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  3. 2016V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentationcited by 2,292
  4. 2015Unsupervised Deep Embedding for Clustering Analysiscited by 767
  5. 2018Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challengecited by 511
  6. 2021FedProto: Federated Prototype Learning across Heterogeneous Clientscited by 352
  7. 20183D MRI brain tumor segmentation using autoencoder regularizationcited by 328
  8. 2019Federated Evaluation of On-device Personalizationcited by 147
  9. 2020Federated Visual Classification with Real-World Data Distributioncited by 128
  10. 2023Multimodal Federated Learning via Contrastive Representation Ensemblecited by 86
  11. 2019Hetero-Modal Variational Encoder-Decoder for Joint Modality Completion and Segmentationcited by 68
  12. 2021Knowledge distillation from multi-modal to mono-modal segmentation networkscited by 68
10 more
  1. 2021ACN: Adversarial Co-training Network for Brain Tumor Segmentation with Missing Modalitiescited by 43
  2. 2019Brain Tumor Segmentation on MRI with Missing Modalitiescited by 42
  3. 2021Latent Correlation Representation Learning for Brain Tumor Segmentation with Missing MRI Modalitiescited by 42
  4. 2021Multimodal Federated Learning on IoT Datacited by 32
  5. 2020Experiments of Federated Learning for COVID-19 Chest X-ray Imagescited by 31
  6. 2022SMU-Net: Style matching U-Net for brain tumor segmentation with missing modalitiescited by 31
  7. 2021Multi-Anchor Active Domain Adaptation for Semantic Segmentationcited by 29
  8. 2019One-pass Multi-task Networks with Cross-task Guided Attention for Brain Tumor Segmentationcited by 25
  9. 2021FedDis: Disentangled Federated Learning for Unsupervised Brain Pathology Segmentationcited by 19
  10. 2021A Unified Framework for Generalized Low-Shot Medical Image Segmentation with Scarce Datacited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 3.

8 papers.

  1. 2026Federated Modality-specific Encoders and Partially Personalized Fusion Decoder for Multimodal Brain Tumor Segmentationcited by 3
  2. 2025Hyper-modal Imputation Diffusion Embedding with Dual-Distillation for Federated Multimodal Knowledge Graph Completioncited by 1
  3. 2026Med-MMFL: A Multimodal Federated Learning Benchmark in Healthcarecited by 1
  4. 2025Multimodal Federated Learning: A Survey through the Lens of Different FL Paradigmscited by 0
  5. 2025A Scenario-Oriented Survey of Federated Recommender Systems: Techniques, Challenges, and Future Directionscited by 0
  6. 2025Mix-modal Federated Learning for MRI Image Segmentationcited by 0
  7. 2025HybridVFL: Disentangled Feature Learning for Edge-Enabled Vertical Federated Multimodal Classificationcited by 0
  8. 2026Multimodal Federated Learning under Dual-Axis Modality Missingnesscited by 0