# LLM-driven Medical Report Generation via Communication-efficient Heterogeneous Federated Learning

arXiv: https://arxiv.org/abs/2506.17562 · HTML: https://mlsanity.com/paper/2506.17562/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 51 of its 72 references that are in the map.
- First posted on arXiv 21 Jun 2025.

## Builds on (most cited first, 30 of 51)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [The Future of Digital Health with Federated Learning](https://mlsanity.com/paper/2003.08119/) (2020)
- [MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs](https://mlsanity.com/paper/1901.07042/) (2019)
- [Generating Radiology Reports via Memory-driven Transformer](https://mlsanity.com/paper/2010.16056/) (2020)
- [CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT](https://mlsanity.com/paper/2004.09167/) (2020)
- [VeRA: Vector-based Random Matrix Adaptation](https://mlsanity.com/paper/2310.11454/) (2023)
- [Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Models](https://mlsanity.com/paper/2305.18292/) (2023)
- [On the Automatic Generation of Medical Imaging Reports](https://mlsanity.com/paper/1711.08195/) (2017)
- [Towards Building the Federated GPT: Federated Instruction Tuning](https://mlsanity.com/paper/2305.05644/) (2023)
- [Exploring and Distilling Posterior and Prior Knowledge for Radiology Report Generation](https://mlsanity.com/paper/2106.06963/) (2021)
- [Improving LoRA in Privacy-preserving Federated Learning](https://mlsanity.com/paper/2403.12313/) (2024)
- [When Radiology Report Generation Meets Knowledge Graph](https://mlsanity.com/paper/2002.08277/) (2020)
- [FedPara: Low-Rank Hadamard Product for Communication-Efficient Federated Learning](https://mlsanity.com/paper/2108.06098/) (2021)
- [Interactive and Explainable Region-guided Radiology Report Generation](https://mlsanity.com/paper/2304.08295/) (2023)
- [PromptFL: Let Federated Participants Cooperatively Learn Prompts Instead of Models -- Federated Learning in Age of Foundation Model](https://mlsanity.com/paper/2208.11625/) (2022)
- [Dynamic Graph Enhanced Contrastive Learning for Chest X-ray Report Generation](https://mlsanity.com/paper/2303.10323/) (2023)
- [Multi-Modal Domain Adaptation for Fine-Grained Action Recognition](https://mlsanity.com/paper/2001.09691/) (2020)
- [R2GenGPT: Radiology Report Generation with Frozen LLMs](https://mlsanity.com/paper/2309.09812/) (2023)
- [METransformer: Radiology Report Generation by Transformer with Multiple Learnable Expert Tokens](https://mlsanity.com/paper/2304.02211/) (2023)

## Built on by (most cited first, 4 of 4)

- [A Chain of Diagnosis Framework for Accurate and Explainable Radiology Report Generation](https://mlsanity.com/paper/2508.09566/) (2025)
- [RANGER: Sparsely-Gated Mixture-of-Experts with Adaptive Retrieval Re-ranking for Pathology Report Generation](https://mlsanity.com/paper/2603.04348/) (2026)
- [Federated Fine-Tuning of Sparsely-Activated Large Language Models on Resource-Constrained Devices](https://mlsanity.com/paper/2508.19078/) (2025)
- [Semantic Context-aware mOdality fUsion Transformer (SCOUT): A Context-Aware Multimodal Transformer for Concept-Grounded Pathology Report Generation](https://mlsanity.com/paper/2605.01144/) (2026)

## Abstract (arXiv)

LLMs have demonstrated significant potential in Medical Report Generation (MRG), yet their development requires large amounts of medical image-report pairs, which are commonly scattered across multiple centers. Centralizing these data is exceptionally challenging due to privacy regulations, thereby impeding model development and broader adoption of LLM-driven MRG models. To address this challenge, we present FedMRG, the first framework that leverages Federated Learning (FL) to enable privacy-preserving, multi-center development of LLM-driven MRG models, specifically designed to overcome the critical challenge of communication-efficient LLM training under multi-modal data heterogeneity. To start with, our framework tackles the fundamental challenge of communication overhead in FL-LLM tuning by employing low-rank factorization to efficiently decompose parameter updates, significantly reducing gradient transmission costs and making LLM-driven MRG feasible in bandwidth-constrained FL settings. Furthermore, we observed the dual heterogeneity in MRG under the FL scenario: varying image characteristics across medical centers, as well as diverse reporting styles and terminology preferences. To address this, we further enhance FedMRG with (1) client-aware contrastive learning in the MRG encoder, coupled with diagnosis-driven prompts, which capture both globally generalizable and locally distinctive features while maintaining diagnostic accuracy; and (2) a dual-adapter mutual boosting mechanism in the MRG decoder that harmonizes generic and specialized adapters to address variations in reporting styles and terminology. Through extensive evaluation of our established FL-MRG benchmark, we demonstrate the generalizability and adaptability of FedMRG, underscoring its potential in harnessing multi-center data and generating clinically accurate reports while maintaining communication efficiency.