# MAFM^3: Modular Adaptation of Foundation Models for Multi-Modal Medical AI

arXiv: https://arxiv.org/abs/2511.11212 · HTML: https://mlsanity.com/paper/2511.11212/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 41 references that are in the map.
- First posted on arXiv 14 Nov 2025.

## Builds on (most cited first, 25 of 25)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://mlsanity.com/paper/2006.16668/) (2020)
- [A Survey on Deep Learning in Medical Image Analysis](https://mlsanity.com/paper/1702.05747/) (2017)
- [UNETR: Transformers for 3D Medical Image Segmentation](https://mlsanity.com/paper/2103.10504/) (2021)
- [MedCLIP: Contrastive Learning from Unpaired Medical Images and Text](https://mlsanity.com/paper/2210.10163/) (2022)
- [MONAI: An open-source framework for deep learning in healthcare](https://mlsanity.com/paper/2211.02701/) (2022)
- [Making the Most of Text Semantics to Improve Biomedical Vision--Language Processing](https://mlsanity.com/paper/2204.09817/) (2022)
- [Generalist Foundation Models from a Multimodal Dataset for 3D Computed Tomography](https://mlsanity.com/paper/2403.17834/) (2024)
- [Self-Expansion of Pre-trained Models with Mixture of Adapters for Continual Learning](https://mlsanity.com/paper/2403.18886/) (2024)
- [Deep Neural Networks for Survival Analysis Based on a Multi-Task Framework](https://mlsanity.com/paper/1801.05512/) (2018)
- [MOS: Model Surgery for Pre-Trained Model-Based Class-Incremental Learning](https://mlsanity.com/paper/2412.09441/) (2024)
- [Overview of the HECKTOR Challenge at MICCAI 2021: Automatic Head and Neck Tumor Segmentation and Outcome Prediction in PET/CT Images](https://mlsanity.com/paper/2201.04138/) (2022)
- [Adapter Learning in Pretrained Feature Extractor for Continual Learning of Diseases](https://mlsanity.com/paper/2304.09042/) (2023)
- [A Survey on Trustworthiness in Foundation Models for Medical Image Analysis](https://mlsanity.com/paper/2407.15851/) (2024)
- [Low-Rank Mixture-of-Experts for Continual Medical Image Segmentation](https://mlsanity.com/paper/2406.13583/) (2024)
- [A Foundation Model for Brain Lesion Segmentation with Mixture of Modality Experts](https://mlsanity.com/paper/2405.10246/) (2024)
- [SurvRNC: Learning Ordered Representations for Survival Prediction using Rank-N-Contrast](https://mlsanity.com/paper/2403.10603/) (2024)
- [Automatic Segmentation of Head and Neck Tumor: How Powerful Transformers Are?](https://mlsanity.com/paper/2201.06251/) (2022)
- [DynaMMo: Dynamic Model Merging for Efficient Class Incremental Learning for Medical Images](https://mlsanity.com/paper/2404.14099/) (2024)

## Abstract (arXiv)

Foundational models are trained on extensive datasets to capture the general trends of a domain. However, in medical imaging, the scarcity of data makes pre-training for every domain, modality, or task challenging. Instead of building separate models, we propose MAFM^3 (Modular Adaptation of Foundation Models for Multi-Modal Medical AI), a framework that enables a single foundation model to expand into diverse domains, tasks, and modalities through lightweight modular components. These components serve as specialized skill sets that allow the system to flexibly activate the appropriate capability at the inference time, depending on the input type or clinical objective. Unlike conventional adaptation methods that treat each new task or modality in isolation, MAFM^3 provides a unified and expandable framework for efficient multitask and multimodality adaptation. Empirically, we validate our approach by adapting a chest CT foundation model initially trained for classification into prognosis and segmentation modules. Our results show improved performance on both tasks. Furthermore, by incorporating PET scans, MAFM^3 achieved an improvement in the Dice score 5% compared to the respective baselines. These findings establish that foundation models, when equipped with modular components, are not inherently constrained to their initial training scope but can evolve into multitask, multimodality systems for medical imaging. The code implementation of this work can be found at https://github.com/Areeb2735/CTscan_prognosis_VLM