mlsanityData to 14 Oct 2026

Gla-AI4BioMed at RRG24: Visual Instruction-tuned Adaptation for Radiology Report Generation

arXiv 2412.04954 ↗First posted 6 Dec 2024cs.CV cs.CL cs.LGcited by 4

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

We introduce a radiology-focused visual language model designed to generate radiology reports from chest X-rays. Building on previous findings that large language models (LLMs) can acquire multimodal capabilities when aligned with pretrained vision encoders, we demonstrate similar potential with chest X-ray images. This integration enhances the ability of model to understand and describe chest X-ray images. Our model combines an image encoder with a fine-tuned LLM based on the Vicuna-7B architecture, enabling it to generate different sections of a radiology report with notable accuracy. The training process involves a two-stage approach: (i) initial alignment of chest X-ray features with the LLM (ii) followed by fine-tuning for radiology report generation.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 24 of its 43 references that are in the map. First posted on arXiv 6 Dec 2024.

Builds on

24 of 43 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day1 Jun

Other references · 23

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  5. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  6. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  7. 2013Generating Sequences With Recurrent Neural Networkscited by 554
  8. 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 392
  9. 2020CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERTcited by 337
  10. 2019PadChest: A large chest x-ray image dataset with multi-label annotated reportscited by 309
  11. 2023Learning to Exploit Temporal Structure for Biomedical Vision-Language Processingcited by 211
  12. 2020Improving Factual Completeness and Consistency of Image-to-Text Radiology Report Generationcited by 141
11 more
  1. 2024CheXpert Plus: Augmenting a Large Chest X-ray Dataset with Text Radiology Reports, Patient Demographics and Additional Image Formatscited by 106
  2. 2022Improving Chest X-Ray Report Generation by Leveraging Warm Startingcited by 95
  3. 2023PromptMRG: Diagnosis-Driven Prompts for Medical Report Generationcited by 95
  4. 2022Improving the Factual Correctness of Radiology Report Generation with Semantic Rewardscited by 83
  5. 2020BIMCV COVID-19+: a large annotated dataset of RX and CT images from COVID-19 patientscited by 82
  6. 2023ELIXR: Towards a general purpose X-ray artificial intelligence system through alignment of large language models and radiology vision encoderscited by 50
  7. 2023Radiology-GPT: A Large Language Model for Radiologycited by 44
  8. 2023Multimodal Image-Text Matching Improves Retrieval-based Chest X-Ray Report Generationcited by 37
  9. 2023Style-Aware Radiology Report Generation with RadGraph and Few-Shot Promptingcited by 29
  10. 2024Overview of the First Shared Task on Clinical Text Generation: RRG24 and "Discharge Me!"cited by 20
  11. 2024CLCE: An Approach to Refining Cross-Entropy and Contrastive Learning for Optimized Learning Fusioncited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Route, Retrieve, Reflect, Repair: Self-Improving Agentic Framework for Visual Detection and Linguistic Reasoning in Medical Imagingcited by 0
  2. 2026Ran Score: a LLM-based Evaluation Score for Radiology Report Generationcited by 0
  3. 2026Can Multimodal Large Language Models Understand Pathologic Movements? A Pilot Study on Seizure Semiologycited by 0
  4. 2026Reporting Practice Matters: The Impact of Reference Choice on Chest X-ray Report Evaluationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023MAIRA-1: A specialised large multimodal model for radiology report generation
  2. 2025A Generative Framework for Bidirectional Image-Report Understanding in Chest Radiography
  3. 2026RadJEPA: Radiology Encoder for Chest X-Rays via Joint Embedding Predictive Architecture
  4. 2025Evaluating Vision Language Model Adaptations for Radiology Report Generation in Low-Resource Languages
  5. 2024Multimodal Large Language Model driven Radiology Report Generation with Clinical Knowledge Enhancement
  6. 2025Exploring the Capabilities of Large Language Model Encoders for Image-Text Retrieval in Chest X-rays
  7. 2025RadAlign: Advancing Radiology Report Generation with Vision-Language Concept Alignment
  8. 2023MedXChat: A Unified Multimodal Large Language Model Framework towards CXRs Understanding and Generation