# M4oE: A Foundation Model for Medical Multimodal Image Segmentation with Mixture of Experts

arXiv: https://arxiv.org/abs/2405.09446 · HTML: https://mlsanity.com/paper/2405.09446/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Placed in Medical and biomedical MoE, one of 12 directions in Mixture-of-experts.
- Builds on 17 of its 20 references that are in the map.
- First posted on arXiv 15 May 2024.

## Placed in

- Mixture-of-experts / [Medical and biomedical MoE](https://mlsanity.com/moe/medical-and-biomedical-moe/)

## Builds on (most cited first, 17 of 17)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation](https://mlsanity.com/paper/2105.05537/) (2021)
- [MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs](https://mlsanity.com/paper/1901.07042/) (2019)
- [AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation](https://mlsanity.com/paper/2206.08023/) (2022)
- [UNet 3+: A Full-Scale Connected UNet for Medical Image Segmentation](https://mlsanity.com/paper/2004.08790/) (2020)
- [Med3D: Transfer Learning for 3D Medical Image Analysis](https://mlsanity.com/paper/1904.00625/) (2019)
- [SAM-Med2D](https://mlsanity.com/paper/2308.16184/) (2023)
- [STU-Net: Scalable and Transferable Medical Image Segmentation Models Empowered by Large-Scale Supervised Pre-training](https://mlsanity.com/paper/2304.06716/) (2023)
- [Medical Image Analysis using Convolutional Neural Networks: A Review](https://mlsanity.com/paper/1709.02250/) (2017)
- [DoDNet: Learning to segment multi-organ and tumors from multiple partially labeled datasets](https://mlsanity.com/paper/2011.10217/) (2020)
- [Swin MAE: Masked Autoencoders for Small Datasets](https://mlsanity.com/paper/2212.13805/) (2022)
- [Perspectives and Prospects on Transformer Architecture for Cross-Modal Tasks with Language and Vision](https://mlsanity.com/paper/2103.04037/) (2021)
- [Universal Segmentation of 33 Anatomies](https://mlsanity.com/paper/2203.02098/) (2022)
- [Combining Self-Training and Hybrid Architecture for Semi-supervised Abdominal Organ Segmentation](https://mlsanity.com/paper/2207.11512/) (2022)
- [Towards the Unification of Generative and Discriminative Visual Foundation Model: A Survey](https://mlsanity.com/paper/2312.10163/) (2023)

## Built on by (most cited first, 26 of 26)

- [LiMoE: Mixture of LiDAR Representation Learners from Automotive Scenes](https://mlsanity.com/paper/2501.04004/) (2025)
- [Medical Multimodal Foundation Models in Clinical Diagnosis and Treatment: Applications, Challenges, and Future Directions](https://mlsanity.com/paper/2412.02621/) (2024)
- [Mixture of Experts (MoE): A Big Data Perspective](https://mlsanity.com/paper/2501.16352/) (2025)
- [CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Models](https://mlsanity.com/paper/2503.07667/) (2025)
- [MAUP: Training-free Multi-center Adaptive Uncertainty-aware Prompting for Cross-domain Few-shot Medical Image Segmentation](https://mlsanity.com/paper/2508.03511/) (2025)
- [MoE3D: Mixture of Experts meets Multi-Modal 3D Understanding](https://mlsanity.com/paper/2511.22103/) (2025)
- [Seg2Change: Adapting Open-Vocabulary Semantic Segmentation Model for Remote Sensing Change Detection](https://mlsanity.com/paper/2604.11231/) (2026)
- [Mamba Goes HoME: Hierarchical Soft Mixture-of-Experts for 3D Medical Image Segmentation](https://mlsanity.com/paper/2507.06363/) (2025)
- [A Multimodal Foundation Model to Enhance Generalizability and Data Efficiency for Pan-cancer Prognosis Prediction](https://mlsanity.com/paper/2509.12600/) (2025)
- [Multimodal LLM With Hierarchical Mixture-of-Experts for VQA on 3D Brain MRI](https://mlsanity.com/paper/2509.25889/) (2025)
- [REN: Anatomically-Informed Mixture-of-Experts for Interstitial Lung Disease Diagnosis](https://mlsanity.com/paper/2510.04923/) (2025)
- [Understanding the Transfer Limits of Vision Foundation Models](https://mlsanity.com/paper/2601.15888/) (2026)
- [The Rise of Sparse Mixture-of-Experts: A Survey from Algorithmic Foundations to Decentralized Architectures and Vertical Domain Applications](https://mlsanity.com/paper/2602.08019/) (2026)
- [SegMoTE: Token-Level Mixture of Experts for Medical Image Segmentation](https://mlsanity.com/paper/2602.19213/) (2026)
- [Distribution-aware Fairness Learning in Medical Image Segmentation From A Control-Theoretic Perspective](https://mlsanity.com/paper/2502.00619/) (2025)
- [MicarVLMoE: A Modern Gated Cross-Aligned Vision-Language Mixture of Experts Model for Medical Image Captioning and Report Generation](https://mlsanity.com/paper/2504.20343/) (2025)
- [Boundary-Aware Vision Transformer for Angiography Vascular Network Segmentation](https://mlsanity.com/paper/2506.12980/) (2025)
- [ALL-PET: A Low-resource and Low-shot PET Foundation Model in Projection Domain](https://mlsanity.com/paper/2509.09130/) (2025)
- [SHMoAReg: Spark Deformable Image Registration via Spatial Heterogeneous Mixture of Experts and Attention Heads](https://mlsanity.com/paper/2509.20073/) (2025)
- [A Foundation Model for Brain MRI with Dynamic Modality Integration](https://mlsanity.com/paper/2511.03014/) (2025)
- [Interpretable Alzheimer's Diagnosis via Multimodal Fusion of Regional Brain Experts](https://mlsanity.com/paper/2512.10966/) (2025)
- [Domain-Expert-Guided Hybrid Mixture-of-Experts for Medical AI: Integrating Data-Driven Learning with Clinical Priors](https://mlsanity.com/paper/2601.17977/) (2026)
- [Tackling Multimodal Learning Challenges with Mixture-of-Expert: A Survey](https://mlsanity.com/paper/2605.27431/) (2026)
- [MoPET: Parameter-Efficient Mixture-of-Experts for Unified Medical Image Classification](https://mlsanity.com/paper/2607.29462/) (2026)
- [LocAnyMed: Vision-Language Grounding for Multimodal Medical Images](https://mlsanity.com/paper/2608.03322/) (2026)
- [Generalist-Specialist Mixture-of-Experts for Rare Pathology Detection in Multimodal Imaging](https://mlsanity.com/paper/2609.18688/) (2026)

## Abstract (arXiv)

Medical imaging data is inherently heterogeneous across different modalities and clinical centers, posing unique challenges for developing generalizable foundation models. Conventional entails training distinct models per dataset or using a shared encoder with modality-specific decoders. However, these approaches incur heavy computational overheads and suffer from poor scalability. To address these limitations, we propose the Medical Multimodal Mixture of Experts (M4oE) framework, leveraging the SwinUNet architecture. Specifically, M4oE comprises modality-specific experts; each separately initialized to learn features encoding domain knowledge. Subsequently, a gating network is integrated during fine-tuning to modulate each expert's contribution to the collective predictions dynamically. This enhances model interpretability and generalization ability while retaining expertise specialization. Simultaneously, the M4oE architecture amplifies the model's parallel processing capabilities, and it also ensures the model's adaptation to new modalities with ease. Experiments across three modalities reveal that M4oE can achieve 3.45% over STU-Net-L, 5.11% over MED3D, and 11.93% over SAM-Med2D across the MICCAI FLARE22, AMOS2022, and ATLAS2023 datasets. Moreover, M4oE showcases a significant reduction in training duration with 7 hours less while maintaining a parameter count that is only 30% of its compared methods. The code is available at https://github.com/JefferyJiang-YF/M4oE.