# FEDKIM: Adaptive Federated Knowledge Injection into Medical Foundation Models

arXiv: https://arxiv.org/abs/2408.10276 · HTML: https://mlsanity.com/paper/2408.10276/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 76 references that are in the map.
- First posted on arXiv 17 Aug 2024.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day](https://mlsanity.com/paper/2306.00890/) (2023)
- [Adaptive Federated Optimization](https://mlsanity.com/paper/2003.00295/) (2020)
- [Personalized Federated Learning with Moreau Envelopes](https://mlsanity.com/paper/2006.08848/) (2020)
- [LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model](https://mlsanity.com/paper/2304.15010/) (2023)
- [SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answering](https://mlsanity.com/paper/2102.09542/) (2021)
- [A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPT](https://mlsanity.com/paper/2302.09419/) (2023)
- [ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledge](https://mlsanity.com/paper/2303.14070/) (2023)
- [Improving Federated Learning Personalization via Model Agnostic Meta Learning](https://mlsanity.com/paper/1909.12488/) (2019)
- [Multimodal Foundation Models: From Specialists to General-Purpose Assistants](https://mlsanity.com/paper/2309.10020/) (2023)
- [BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Tasks](https://mlsanity.com/paper/2305.17100/) (2023)
- [MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Experts](https://mlsanity.com/paper/2404.15159/) (2024)
- [Towards Building Multilingual Language Model for Medicine](https://mlsanity.com/paper/2402.13963/) (2024)
- [LLaVA-MoLE: Sparse Mixture of LoRA Experts for Mitigating Data Conflicts in Instruction Finetuning MLLMs](https://mlsanity.com/paper/2401.16160/) (2024)
- [FedCLIP: Fast Generalization and Personalization for CLIP in Federated Learning](https://mlsanity.com/paper/2302.13485/) (2023)
- [Input Augmentation with SAM: Boosting Medical Image Segmentation with Segmentation Foundation Model](https://mlsanity.com/paper/2304.11332/) (2023)
- [ECG-QA: A Comprehensive Question Answering Dataset Combined With Electrocardiogram](https://mlsanity.com/paper/2306.15681/) (2023)
- [Foundation Metrics for Evaluating Effectiveness of Healthcare Conversations Powered by Generative AI](https://mlsanity.com/paper/2309.12444/) (2023)
- [Yet Another ICU Benchmark: A Flexible Multi-Center Framework for Clinical ML](https://mlsanity.com/paper/2306.05109/) (2023)
- [Mini-Model Adaptation: Efficiently Extending Pretrained Models to New Languages via Aligned Shallow Training](https://mlsanity.com/paper/2212.10503/) (2022)
- [FedPFT: Federated Proxy Fine-Tuning of Foundation Models](https://mlsanity.com/paper/2404.11536/) (2024)
- [Adversarial Robustness through Bias Variance Decomposition: A New Perspective for Federated Learning](https://mlsanity.com/paper/2009.09026/) (2020)
- [Mutual Enhancement of Large and Small Language Models with Cross-Silo Knowledge Transfer](https://mlsanity.com/paper/2312.05842/) (2023)
- [Communication-Compressed Adaptive Gradient Method for Distributed Nonconvex Optimization](https://mlsanity.com/paper/2111.00705/) (2021)

## Built on by (most cited first, 5 of 5)

- [Ten Challenging Problems in Federated Foundation Models](https://mlsanity.com/paper/2502.12176/) (2025)
- [Vision Foundation Models in Medical Image Analysis: Advances and Challenges](https://mlsanity.com/paper/2502.14584/) (2025)
- [Federated EndoViT: Pretraining Vision Transformers via Federated Learning on Endoscopic Image Collections](https://mlsanity.com/paper/2504.16612/) (2025)
- [Asymmetrical Reciprocity-based Federated Learning for Resolving Disparities in Medical Diagnosis](https://mlsanity.com/paper/2412.19654/) (2024)
- [MEDMKG: Benchmarking Medical Knowledge Exploitation with Multimodal Knowledge Graph](https://mlsanity.com/paper/2505.17214/) (2025)

## Abstract (arXiv)

Foundation models have demonstrated remarkable capabilities in handling diverse modalities and tasks, outperforming conventional artificial intelligence (AI) approaches that are highly task-specific and modality-reliant. In the medical domain, however, the development of comprehensive foundation models is constrained by limited access to diverse modalities and stringent privacy regulations. To address these constraints, this study introduces a novel knowledge injection approach, FedKIM, designed to scale the medical foundation model within a federated learning framework. FedKIM leverages lightweight local models to extract healthcare knowledge from private data and integrates this knowledge into a centralized foundation model using a designed adaptive Multitask Multimodal Mixture Of Experts (M3OE) module. This method not only preserves privacy but also enhances the model's ability to handle complex medical tasks involving multiple modalities. Our extensive experiments across twelve tasks in seven modalities demonstrate the effectiveness of FedKIM in various settings, highlighting its potential to scale medical foundation models without direct access to sensitive data.