mlsanityData to 14 Oct 2026

FM2: Unified Federated Foundation Models for Heterogeneous Multimodal Medical Imaging

arXiv 2607.13386 ↗First posted 15 Jul 2026cs.CVcited by 0

On the map: Mixture-of-experts → Medical and biomedical MoE

Abstract (arXiv)

Building foundation models for medical imaging requires pooling data across institutions, yet privacy regulations prohibit centralized aggregation. Existing Federated Foundation Models either fine-tune natural-image models with poor medical-domain transfer, or train from scratch within a single modality, lacking the flexibility to unify tasks. We identify an under-explored challenge, Imaging Modality Heterogeneity, where clients operate under two structural regimes: Overlapped (shared modalities with heterogeneous label distributions) and Non-overlapped (fully disjoint modalities per client). We propose FM2, a unified framework that trains the core backbone from scratch to preserve medical domain fidelity while optionally incorporating biomedical pretrained encoders for vision-language alignment. FM2 equips each client with dual Mixture-of-Experts modules (a Class-wise MoE for personalized category knowledge and a Domain-wise MoE for shared cross-modality representations), coupled with a Heterogeneous Modality Alignment (HMA) regularizer that explicitly aligns modality-specific expert parameters, admitting provable O(1/√T) convergence and generalization guarantees. FM2 further incorporates Caption-Enhanced Learning (CEL), where locally retained GPT-4o-generated captions serve as a textual semantic bridge enabling representation transfer across clients with disjoint modalities, and demonstrates extensibility to Federated Medical VQA. Experiments on our MIMH benchmark (classification and CEL) and real-world medical VQA datasets confirm consistent superiority over state-of-the-art federated baselines and strong out-of-modality generalization across all three tasks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Medical and biomedical MoE, one of 12 directions in Mixture-of-experts. Builds on 30 of its 56 references that are in the map. First posted on arXiv 15 Jul 2026.

Builds on

30 of 56 references.

References · 30

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  5. 2023Segment Anythingcited by 8,234
  6. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  8. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  9. 2020Adaptive Federated Optimizationcited by 1,046
  10. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 1,009
  11. 2020Ditto: Fair and Robust Federated Learning Through Personalizationcited by 672
  12. 2021FedBN: Federated Learning on Non-IID Features via Local Batch Normalizationcited by 571
18 more
  1. 2021Federated Learning Based on Dynamic Regularizationcited by 562
  2. 2019Federated Learning with Personalization Layerscited by 541
  3. 2021Exploiting Shared Representations for Personalized Federated Learningcited by 541
  4. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 449
  5. 2022Multimodal Learning with Transformers: A Surveycited by 400
  6. 2021FedProto: Federated Prototype Learning across Heterogeneous Clientscited by 352
  7. 2020Personalized Federated Learning: A Meta-Learning Approachcited by 312
  8. 2020Personalized Federated Learning with First Order Model Optimizationcited by 201
  9. 2021Federated Multi-Task Learning under a Mixture of Distributionscited by 192
  10. 2022PromptFL: Let Federated Participants Cooperatively Learn Prompts Instead of Models -- Federated Learning in Age of Foundation Modelcited by 134
  11. 2020Lower Bounds and Optimal Algorithms for Personalized Federated Learningcited by 123
  12. 2023Prompt Federated Learning for Weather Forecasting: Toward Foundation Models on Meteorological Datacited by 40
  13. 2023Strong Baselines for Parameter Efficient Few-Shot Fine-tuningcited by 37
  14. 2023Bold but Cautious: Unlocking the Potential of Personalized Federated Learning through Cautiously Aggressive Collaborationcited by 23
  15. 2024Personalized Adapter for Large Meteorology Model on Devices: Towards Weather Foundation Modelscited by 15
  16. 2024Federated Foundation Models on Heterogeneous Time Seriescited by 13
  17. 2025FeDaL: Federated Dataset Learning for General Time Series Foundation Modelscited by 10
  18. 2026OmniFM: Toward Modality-Robust and Task-Agnostic Federated Learning for Heterogeneous Medical Imagingcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Distributionally Robust Alignment for Medical Federated Vision-Language Pre-training Under Data Heterogeneity
  2. 2025FedSKD: Aggregation-free Model-heterogeneous Federated Learning via Multi-dimensional Similarity Knowledge Distillation for Medical Image Classification
  3. 2026MedMix: Specialization-Consistent Federated Sparse MoEs under Modality Heterogeneity
  4. 2023Federated Learning for Data and Model Heterogeneity in Medical Imaging
  5. 2024Med-MoE: Mixture of Domain-Specific Experts for Lightweight Medical Vision-Language Models
  6. 2025Adapting Foundation Vision-Language Models to Medical Diagnosis via Query-Driven Expert Bridging
  7. 2024M4oE: A Foundation Model for Medical Multimodal Image Segmentation with Mixture of Experts
  8. 2026DSFedMed: Dual-Scale Federated Medical Image Segmentation via Mutual Distillation Between Foundation and Lightweight Models