# Gla-AI4BioMed at RRG24: Visual Instruction-tuned Adaptation for Radiology Report Generation

arXiv: https://arxiv.org/abs/2412.04954 · HTML: https://mlsanity.com/paper/2412.04954/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 24 of its 43 references that are in the map.
- First posted on arXiv 6 Dec 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)

## Builds on (most cited first, 24 of 24)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [PaLM-E: An Embodied Multimodal Language Model](https://mlsanity.com/paper/2303.03378/) (2023)
- [LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day](https://mlsanity.com/paper/2306.00890/) (2023)
- [ZeRO: Memory Optimizations Toward Training Trillion Parameter Models](https://mlsanity.com/paper/1910.02054/) (2019)
- [Generating Sequences With Recurrent Neural Networks](https://mlsanity.com/paper/1308.0850/) (2013)
- [Med-Flamingo: a Multimodal Medical Few-shot Learner](https://mlsanity.com/paper/2307.15189/) (2023)
- [CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT](https://mlsanity.com/paper/2004.09167/) (2020)
- [PadChest: A large chest x-ray image dataset with multi-label annotated reports](https://mlsanity.com/paper/1901.07441/) (2019)
- [Learning to Exploit Temporal Structure for Biomedical Vision-Language Processing](https://mlsanity.com/paper/2301.04558/) (2023)
- [Improving Factual Completeness and Consistency of Image-to-Text Radiology Report Generation](https://mlsanity.com/paper/2010.10042/) (2020)
- [CheXpert Plus: Augmenting a Large Chest X-ray Dataset with Text Radiology Reports, Patient Demographics and Additional Image Formats](https://mlsanity.com/paper/2405.19538/) (2024)
- [Improving Chest X-Ray Report Generation by Leveraging Warm Starting](https://mlsanity.com/paper/2201.09405/) (2022)
- [PromptMRG: Diagnosis-Driven Prompts for Medical Report Generation](https://mlsanity.com/paper/2308.12604/) (2023)
- [Improving the Factual Correctness of Radiology Report Generation with Semantic Rewards](https://mlsanity.com/paper/2210.12186/) (2022)
- [BIMCV COVID-19+: a large annotated dataset of RX and CT images from COVID-19 patients](https://mlsanity.com/paper/2006.01174/) (2020)
- [ELIXR: Towards a general purpose X-ray artificial intelligence system through alignment of large language models and radiology vision encoders](https://mlsanity.com/paper/2308.01317/) (2023)
- [Radiology-GPT: A Large Language Model for Radiology](https://mlsanity.com/paper/2306.08666/) (2023)
- [Multimodal Image-Text Matching Improves Retrieval-based Chest X-Ray Report Generation](https://mlsanity.com/paper/2303.17579/) (2023)
- [Style-Aware Radiology Report Generation with RadGraph and Few-Shot Prompting](https://mlsanity.com/paper/2310.17811/) (2023)
- [Overview of the First Shared Task on Clinical Text Generation: RRG24 and "Discharge Me!"](https://mlsanity.com/paper/2409.16603/) (2024)
- [CLCE: An Approach to Refining Cross-Entropy and Contrastive Learning for Optimized Learning Fusion](https://mlsanity.com/paper/2402.14551/) (2024)

## Built on by (most cited first, 4 of 4)

- [Route, Retrieve, Reflect, Repair: Self-Improving Agentic Framework for Visual Detection and Linguistic Reasoning in Medical Imaging](https://mlsanity.com/paper/2601.08192/) (2026)
- [Ran Score: a LLM-based Evaluation Score for Radiology Report Generation](https://mlsanity.com/paper/2603.22935/) (2026)
- [Can Multimodal Large Language Models Understand Pathologic Movements? A Pilot Study on Seizure Semiology](https://mlsanity.com/paper/2605.03352/) (2026)
- [Reporting Practice Matters: The Impact of Reference Choice on Chest X-ray Report Evaluation](https://mlsanity.com/paper/2609.19093/) (2026)

## Abstract (arXiv)

We introduce a radiology-focused visual language model designed to generate radiology reports from chest X-rays. Building on previous findings that large language models (LLMs) can acquire multimodal capabilities when aligned with pretrained vision encoders, we demonstrate similar potential with chest X-ray images. This integration enhances the ability of model to understand and describe chest X-ray images. Our model combines an image encoder with a fine-tuned LLM based on the Vicuna-7B architecture, enabling it to generate different sections of a radiology report with notable accuracy. The training process involves a two-stage approach: (i) initial alignment of chest X-ray features with the LLM (ii) followed by fine-tuning for radiology report generation.