# Toward Federated Large Language Models in Medicine: A Parameter-Efficient Framework for Privacy-Preserving, Multi-Institutional Adaptation

arXiv: https://arxiv.org/abs/2601.22124 · HTML: https://mlsanity.com/paper/2601.22124/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 95 references that are in the map.
- First posted on arXiv 29 Jan 2026.

## Builds on (most cited first, 30 of 36)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [Instruction Tuning for Large Language Models: A Survey](https://mlsanity.com/paper/2308.10792/) (2023)
- [Large Language Models Can Be Strong Differentially Private Learners](https://mlsanity.com/paper/2110.05679/) (2021)
- [Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarization](https://mlsanity.com/paper/2309.07430/) (2023)
- [DiLoCo: Distributed Low-Communication Training of Language Models](https://mlsanity.com/paper/2311.08105/) (2023)
- [Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Models](https://mlsanity.com/paper/2401.06432/) (2024)
- [Selective Aggregation for Low-Rank Adaptation in Federated Learning](https://mlsanity.com/paper/2410.01463/) (2024)
- [FDLoRA: Personalized Federated Learning of Large Language Model via Dual LoRA Tuning](https://mlsanity.com/paper/2406.07925/) (2024)
- [Large Language Models to Identify Social Determinants of Health in Electronic Health Records](https://mlsanity.com/paper/2308.06354/) (2023)
- [Large Language Models for Disease Diagnosis: A Scoping Review](https://mlsanity.com/paper/2409.00097/) (2024)
- [Differentially Private Low-Rank Adaptation of Large Language Model Using Federated Learning](https://mlsanity.com/paper/2312.17493/) (2023)
- [Natural Language Processing of Clinical Notes on Chronic Diseases: Systematic Review](https://mlsanity.com/paper/1908.05780/) (2019)
- [Advances and Open Challenges in Federated Foundation Models](https://mlsanity.com/paper/2404.15381/) (2024)
- [PubTator 3.0: an AI-powered Literature Resource for Unlocking Biomedical Knowledge](https://mlsanity.com/paper/2401.11048/) (2024)
- [Fine-Tuning Large Language Models with User-Level Differential Privacy](https://mlsanity.com/paper/2407.07737/) (2024)
- [Federated Large Language Models: Current Progress and Future Directions](https://mlsanity.com/paper/2409.15723/) (2024)
- [BioInstruct: Instruction Tuning of Large Language Models for Biomedical Natural Language Processing](https://mlsanity.com/paper/2310.19975/) (2023)
- [A Comprehensive Survey on Relation Extraction: Recent Advances and New Frontiers](https://mlsanity.com/paper/2306.02051/) (2023)
- [AIONER: All-in-one scheme-based biomedical named entity recognition using deep learning](https://mlsanity.com/paper/2211.16944/) (2022)
- [LLMs are not Zero-Shot Reasoners for Biomedical Information Extraction](https://mlsanity.com/paper/2408.12249/) (2024)
- [Information Extraction from Clinical Notes: Are We Ready to Switch to Large Language Models?](https://mlsanity.com/paper/2411.10020/) (2024)
- [MedINST: Meta Dataset of Biomedical Instructions](https://mlsanity.com/paper/2410.13458/) (2024)
- [Leveraging Open-Source Large Language Models for Clinical Information Extraction in Resource-Constrained Settings](https://mlsanity.com/paper/2507.20859/) (2025)
- [Memorization in Large Language Models in Medicine: Prevalence, Characteristics, and Implications](https://mlsanity.com/paper/2509.08604/) (2025)
- [Distillation of encoder-decoder transformers for sequence labelling](https://mlsanity.com/paper/2302.05454/) (2023)
- [Federated Large Language Models: Feasibility, Robustness, Security and Future Directions](https://mlsanity.com/paper/2505.08830/) (2025)
- [Generalization in Healthcare AI: Evaluation of a Clinical Large Language Model](https://mlsanity.com/paper/2402.10965/) (2024)

## Built on by (most cited first, 1 of 1)

- [TIGER: Inverting Transformer Gradients via Embedding-Subspace Distance Optimization](https://mlsanity.com/paper/2606.18312/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are increasingly adapted for medical applications, but most are trained using data from a single institution because privacy and governance constraints prevent multi-institutional data sharing. As a result, these models often generalize poorly across heterogeneous healthcare systems. We address this gap by introducing Fed-MedLoRA and Fed-MedLoRA+, a parameter-efficient federated framework for collaborative LLM adaptation across healthcare institutions. Fed-MedLoRA transmits only low-rank adapters rather than full model weights, reducing communication overhead. We also evaluate a privacy-preserving variant that applies Gaussian perturbation to transmitted adapter updates. Fed-MedLoRA+ further incorporates adaptive aggregation to better address cross-site heterogeneity in patient populations, annotation practices, and disease distributions. We evaluate the framework on clinical information extraction across five independent patient cohorts totaling 42,198 entities and 41,570 relations, and compare it with zero-shot and fine-tuned LLMs, domain-specific BERT models, and federated baselines. Across all settings, the proposed methods consistently improve extraction performance and generalize better to heterogeneous cohorts. In a real-world case study using clinical notes from the Yale New Haven Health System, the framework demonstrates strong performance under low-resource new-site deployment. These results suggest that federated, parameter-efficient LLM adaptation is feasible, scalable, and effective for multi-institutional clinical deployment.