mlsanityData to 14 Oct 2026

Multimodal Large Language Models for Medical Report Generation via Customized Prompt Tuning

arXiv 2506.15477 ↗First posted 18 Jun 2025cs.CVcited by 5

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Medical report generation from imaging data remains a challenging task in clinical practice. While large language models (LLMs) show great promise in addressing this challenge, their effective integration with medical imaging data still deserves in-depth exploration. In this paper, we present MRG-LLM, a novel multimodal large language model (MLLM) that combines a frozen LLM with a learnable visual encoder and introduces a dynamic prompt customization mechanism. Our key innovation lies in generating instance-specific prompts tailored to individual medical images through conditional affine transformations derived from visual features. We propose two implementations: prompt-wise and promptbook-wise customization, enabling precise and targeted report generation. Extensive experiments on IU X-ray and MIMIC-CXR datasets demonstrate that MRG-LLM achieves state-of-the-art performance in medical report generation. Our code will be made publicly available.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 25 of its 31 references that are in the map. First posted on arXiv 18 Jun 2025.

Builds on

25 of 31 references.

References · 25

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,312
  5. 2014Show and Tell: A Neural Image Caption Generatorcited by 879
  6. 2019MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographscited by 597
  7. 2016Self-critical Sequence Training for Image Captioningcited by 547
  8. 2020Generating Radiology Reports via Memory-driven Transformercited by 368
  9. 2019Meshed-Memory Transformer for Image Captioningcited by 343
  10. 2016Knowing When to Look: Adaptive Attention via A Visual Sentinel for Image Captioningcited by 232
  11. 2022Cross-modal Memory Networks for Radiology Report Generationcited by 185
  12. 2021Exploring and Distilling Posterior and Prior Knowledge for Radiology Report Generationcited by 162
13 more
  1. 2020When Radiology Report Generation Meets Knowledge Graphcited by 145
  2. 2023R2GenGPT: Radiology Report Generation with Frozen LLMscited by 114
  3. 2023PromptMRG: Diagnosis-Driven Prompts for Medical Report Generationcited by 95
  4. 2022Competence-based Multimodal Curriculum Learning for Medical Report Generationcited by 76
  5. 2021Radiology Report Generation with a Learned Knowledge Base and Multi-modal Alignmentcited by 73
  6. 2021Contrastive Attention for Automatic Chest X-ray Report Generationcited by 71
  7. 2022AlignTransformer: Hierarchical Alignment of Visual Regions and Disease Tags for Medical Report Generationcited by 65
  8. 2022Cross-modal Prototype Driven Network for Radiology Report Generationcited by 59
  9. 2022A Medical Semantic-Assisted Transformer for Radiographic Report Generationcited by 44
  10. 2021Progressive Transformer-Based Generation of Radiology Reportscited by 43
  11. 2024HERGen: Elevating Radiology Report Generation with Longitudinal Datacited by 33
  12. 2023Fine-Grained Image-Text Alignment in Medical Imaging Enables Explainable Cyclic Image-Report Generationcited by 26
  13. 2024MedRAT: Unpaired Medical Report Generation via Auxiliary Taskscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 4.

5 papers.

  1. 2026CarePilot: A Multi-Agent Framework for Long-Horizon Computer Task Automation in Healthcarecited by 4
  2. 2025SpiroLLM: Finetuning Pretrained LLMs to Understand Spirogram Time Series with Clinical Validation in COPD Reportingcited by 1
  3. 2026NeuroSymb-MRG: Differentiable Abductive Reasoning with Active Uncertainty Minimization for Radiology Report Generationcited by 1
  4. 2026Towards Real-World Ultrasound Understanding: Large Vision-Language Models from Multi-Image Examinations with Long-Form Reportscited by 0
  5. 2026Toward AI-Assisted Poultry Coccidiosis Diagnosis: Evaluating Gemini and BiomedParse on Eimeria Microscopy Imagescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Effectively Fine-tune to Improve Large Multimodal Models for Radiology Report Generation
  2. 2024PeFoMed: Parameter Efficient Fine-tuning of Multimodal Large Language Models for Medical Imaging
  3. 2023Customizing General-Purpose Foundation Models for Medical Report Generation
  4. 2024Multimodal Large Language Model driven Radiology Report Generation with Clinical Knowledge Enhancement
  5. 2025Exploring the Design Space of 3D MLLMs for CT Report Generation
  6. 2023MedXChat: A Unified Multimodal Large Language Model Framework towards CXRs Understanding and Generation
  7. 2025Med-R2: Perception and Reflection-driven Complex Reasoning for Medical Report Generation
  8. 2025AMRG: Extend Vision Language Models for Automatic Mammography Report Generation