# PeFoMed: Parameter Efficient Fine-tuning of Multimodal Large Language Models for Medical Imaging

arXiv: https://arxiv.org/abs/2401.02797 · HTML: https://mlsanity.com/paper/2401.02797/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 25 of its 54 references that are in the map.
- First posted on arXiv 5 Jan 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)

## Builds on (most cited first, 25 of 25)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day](https://mlsanity.com/paper/2306.00890/) (2023)
- [Med-Flamingo: a Multimodal Medical Few-shot Learner](https://mlsanity.com/paper/2307.15189/) (2023)
- [Generating Radiology Reports via Memory-driven Transformer](https://mlsanity.com/paper/2010.16056/) (2020)
- [LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agents](https://mlsanity.com/paper/2311.05437/) (2023)
- [Exploring and Distilling Posterior and Prior Knowledge for Radiology Report Generation](https://mlsanity.com/paper/2106.06963/) (2021)
- [HuaTuo: Tuning LLaMA Model with Chinese Medical Knowledge](https://mlsanity.com/paper/2304.06975/) (2023)
- [DoctorGLM: Fine-tuning your Chinese Doctor is not a Herculean Task](https://mlsanity.com/paper/2304.01097/) (2023)
- [Multi-Modal Masked Autoencoders for Medical Vision-and-Language Pre-Training](https://mlsanity.com/paper/2209.07098/) (2022)
- [Medical Visual Question Answering: A Survey](https://mlsanity.com/paper/2111.10056/) (2021)
- [AnyMAL: An Efficient and Scalable Any-Modality Augmented Language Model](https://mlsanity.com/paper/2309.16058/) (2023)
- [MedICaT: A Dataset of Medical Images, Captions, and Textual References](https://mlsanity.com/paper/2010.06000/) (2020)
- [Open-Ended Medical Visual Question Answering Through Prefix Tuning of Language Models](https://mlsanity.com/paper/2303.05977/) (2023)
- [LLM-CXR: Instruction-Finetuned LLM for CXR Image Understanding and Generation](https://mlsanity.com/paper/2305.11490/) (2023)
- [Multiple Meta-model Quantifying for Medical Visual Question Answering](https://mlsanity.com/paper/2105.08913/) (2021)
- [Align, Reason and Learn: Enhancing Medical Vision-and-Language Pre-training with Knowledge](https://mlsanity.com/paper/2209.07118/) (2022)
- [ELIXR: Towards a general purpose X-ray artificial intelligence system through alignment of large language models and radiology vision encoders](https://mlsanity.com/paper/2308.01317/) (2023)
- [A Systematic Evaluation of GPT-4V's Multimodal Capability for Medical Image Analysis](https://mlsanity.com/paper/2310.20381/) (2023)
- [MedXChat: A Unified Multimodal Large Language Model Framework towards CXRs Understanding and Generation](https://mlsanity.com/paper/2312.02233/) (2023)
- [Automatic Radiology Report Generation by Learning with Increasingly Hard Negatives](https://mlsanity.com/paper/2305.07176/) (2023)

## Built on by (most cited first, 22 of 22)

- [ErrorRadar: Benchmarking Complex Mathematical Reasoning of Multimodal Large Language Models Via Error Detection](https://mlsanity.com/paper/2410.04509/) (2024)
- [Membership Inference Attacks Against Vision-Language Models](https://mlsanity.com/paper/2501.18624/) (2025)
- [CT-Agent: A Multimodal-LLM Agent for 3D CT Radiology Question Answering](https://mlsanity.com/paper/2505.16229/) (2025)
- [Multimodal Large Language Models for Medicine: A Comprehensive Survey](https://mlsanity.com/paper/2504.21051/) (2025)
- [FedMLLM: Federated Fine-tuning MLLM on Multimodal Heterogeneity Data](https://mlsanity.com/paper/2411.14717/) (2024)
- [Evolution of meta's llama models and parameter-efficient fine-tuning of large language models: a survey](https://mlsanity.com/paper/2510.12178/) (2025)
- [LiteGPT: Large Vision-Language Model for Joint Chest X-ray Localization and Classification Task](https://mlsanity.com/paper/2407.12064/) (2024)
- [Benchmarking Foundation Models and Parameter-Efficient Fine-Tuning for Prognosis Prediction in Medical Imaging](https://mlsanity.com/paper/2506.18434/) (2025)
- [Uncertainty-Driven Expert Control: Enhancing the Reliability of Medical Vision-Language Models](https://mlsanity.com/paper/2507.09209/) (2025)
- [Breaking Failure Cascades: Step-Aware Reinforcement Learning for Medical Multimodal Reasoning](https://mlsanity.com/paper/2606.31825/) (2026)
- [A Multi-Agent System for Complex Reasoning in Radiology Visual Question Answering](https://mlsanity.com/paper/2508.02841/) (2025)
- [Optimizing Multimodal Language Models through Attention-based Interpretability](https://mlsanity.com/paper/2511.23375/) (2025)
- [Decoupling Vision and Language: Codebook Anchored Visual Adaptation](https://mlsanity.com/paper/2602.19449/) (2026)
- [Med-Evo: Test-time Self-evolution for Medical Multimodal Large Language Models](https://mlsanity.com/paper/2603.07443/) (2026)
- [KG-CMI: Knowledge graph enhanced cross-Mamba interaction for medical visual question answering](https://mlsanity.com/paper/2604.00601/) (2026)
- [Retina-RAG: Retrieval-Augmented Vision-Language Modeling for Joint Retinal Diagnosis and Clinical Report Generation](https://mlsanity.com/paper/2605.06173/) (2026)
- [EchoVQA: Enabling Conversational Assistance for Point-of-Care Cardiac Ultrasound](https://mlsanity.com/paper/2605.24159/) (2026)
- [Noise-Aware Visual Representation Learning for Medical Visual Question Answering](https://mlsanity.com/paper/2606.05535/) (2026)
- [Frequency-Domain Dual-Branch Fusion for Medical Visual Question Answering](https://mlsanity.com/paper/2608.08307/) (2026)
- [NeoRed: A Knowledge-Logic-Alignment Multimodal Large Language Model for Neonatal Respiratory Disease Diagnosis](https://mlsanity.com/paper/2609.03527/) (2026)
- [Ranking-Aware Prompt Optimization for Multimodal Clinical Diagnosis](https://mlsanity.com/paper/2609.40361/) (2026)
- [MedPrune: Topology-Efficient Multimodal Multi-Agent Communication Evolution for Medical VQA Tasks](https://mlsanity.com/paper/2610.06695/) (2026)

## Abstract (arXiv)

Multimodal large language models (MLLMs) represent an evolutionary expansion in the capabilities of traditional large language models, enabling them to tackle challenges that surpass the scope of purely text-based applications. It leverages the knowledge previously encoded within these language models, thereby enhancing their applicability and functionality in the reign of multimodal contexts. Recent works investigate the adaptation of MLLMs as a universal solution to address medical multi-modal problems as a generative task. In this paper, we propose a parameter efficient framework for fine-tuning MLLMs, specifically validated on medical visual question answering (Med-VQA) and medical report generation (MRG) tasks, using public benchmark datasets. We also introduce an evaluation metric using the 5-point Likert scale and its weighted average value to measure the quality of the generated reports for MRG tasks, where the scale ratings are labelled by both humans manually and the GPT-4 model. We further assess the consistency of performance metrics across traditional measures, GPT-4, and human ratings for both VQA and MRG tasks. The results indicate that semantic similarity assessments using GPT-4 align closely with human annotators and provide greater stability, yet they reveal a discrepancy when compared to conventional lexical similarity measurements. This questions the reliability of lexical similarity metrics for evaluating the performance of generative models in Med-VQA and report generation tasks. Besides, our fine-tuned model significantly outperforms GPT-4v. This indicates that without additional fine-tuning, multi-modal models like GPT-4v do not perform effectively on medical imaging tasks. The code will be available here: https://github.com/jinlHe/PeFoMed.