# Fine-Tuning MedGemma for Clinical Captioning to Enhance Multimodal RAG over Malaysia CPGs

arXiv: https://arxiv.org/abs/2510.15418 · HTML: https://mlsanity.com/paper/2510.15418/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Medical and biomedical RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 19 of its 36 references that are in the map.
- First posted on arXiv 17 Oct 2025.

## Placed in

- Retrieval-augmented generation / [Medical and biomedical RAG](https://mlsanity.com/rag/medical-and-biomedical-rag/)

## Builds on (most cited first, 19 of 19)

- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day](https://mlsanity.com/paper/2306.00890/) (2023)
- [ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases](https://mlsanity.com/paper/1705.02315/) (2017)
- [MedGemma Technical Report](https://mlsanity.com/paper/2507.05201/) (2025)
- [Towards Expert-Level Medical Question Answering with Large Language Models](https://mlsanity.com/paper/2305.09617/) (2023)
- [An Introduction to Vision-Language Modeling](https://mlsanity.com/paper/2405.17247/) (2024)
- [Vision-Language Models for Medical Report Generation and Visual Question Answering: A Review](https://mlsanity.com/paper/2403.02469/) (2024)
- [MMed-RAG: Versatile Multimodal RAG System for Medical Vision Language Models](https://mlsanity.com/paper/2410.13085/) (2024)
- [RULE: Reliable Multimodal RAG for Factuality in Medical Vision Language Models](https://mlsanity.com/paper/2407.05131/) (2024)
- [Capabilities of GPT-5 on Multimodal Medical Reasoning](https://mlsanity.com/paper/2508.08224/) (2025)
- [Medical Image Captioning via Generative Pretrained Transformers](https://mlsanity.com/paper/2209.13983/) (2022)
- [Describe Anything in Medical Images](https://mlsanity.com/paper/2505.05804/) (2025)
- [RadCLIP: Enhancing Radiologic Image Analysis through Contrastive Language-Image Pre-training](https://mlsanity.com/paper/2403.09948/) (2024)
- [SERPENT-VLM : Self-Refining Radiology Report Generation Using Vision Language Models](https://mlsanity.com/paper/2404.17912/) (2024)
- [Enhancing Health Information Retrieval with RAG by Prioritizing Topical Relevance and Factual Accuracy](https://mlsanity.com/paper/2502.04666/) (2025)
- [Benchmarking GPT-5 for Zero-Shot Multimodal Medical Reasoning in Radiology and Radiation Oncology](https://mlsanity.com/paper/2508.13192/) (2025)

## Built on by (most cited first, 1 of 1)

- [Vision-Language Controlled Deep Unfolding for Joint Medical Image Restoration and Segmentation](https://mlsanity.com/paper/2601.23103/) (2026)

## Abstract (arXiv)

Retrieval-Augmented Generation systems are essential for providing fact-based guidance from Malaysian Clinical Practice Guidelines. However, their effectiveness with image-based queries is limited, as general Vision-Language Model captions often lack clinical specificity and factual grounding. This study proposes and validates a framework to specialize the MedGemma model for generating high-fidelity captions that serve as superior queries. To overcome data scarcity, we employ a knowledge distillation pipeline to create a synthetic dataset across dermatology, fundus, and chest radiography domains, and fine-tune MedGemma using the parameter-efficient QLoRA method. Performance was rigorously assessed through a dual framework measuring both classification accuracy and, via a novel application of the RAGAS framework, caption faithfulness, relevancy, and correctness. The fine-tuned model demonstrated substantial improvements in classification performance, while RAGAS evaluation confirmed significant gains in caption faithfulness and correctness, validating the models ability to produce reliable, factually grounded descriptions. This work establishes a robust pipeline for specializing medical VLMs and validates the resulting model as a high-quality query generator, laying the groundwork for enhancing multimodal RAG systems in evidence-based clinical decision support.