mlsanityData to 7 Oct 2026

Vision-Language Models for Automated Chest X-ray Interpretation: Leveraging ViT and GPT-2

arXiv 2501.12356 ↗First posted 21 Jan 2025cs.CVcited by 2

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Radiology plays a pivotal role in modern medicine due to its non-invasive diagnostic capabilities. However, the manual generation of unstructured medical reports is time consuming and prone to errors. It creates a significant bottleneck in clinical workflows. Despite advancements in AI-generated radiology reports, challenges remain in achieving detailed and accurate report generation. In this study we have evaluated different combinations of multimodal models that integrate Computer Vision and Natural Language Processing to generate comprehensive radiology reports. We employed a pretrained Vision Transformer (ViT-B16) and a SWIN Transformer as the image encoders. The BART and GPT-2 models serve as the textual decoders. We used Chest X-ray images and reports from the IU-Xray dataset to evaluate the usability of the SWIN Transformer-BART, SWIN Transformer-GPT-2, ViT-B16-BART and ViT-B16-GPT-2 models for report generation. We aimed at finding the best combination among the models. The SWIN-BART model performs as the best-performing model among the four models achieving remarkable results in almost all the evaluation metrics like ROUGE, BLEU and BERTScore.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 8 of its 35 references that are in the map. First posted on arXiv 21 Jan 2025.

Builds on

8 of 35 references.

References · 8

  1. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  2. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  3. 2017On the Automatic Generation of Medical Imaging Reportscited by 215
  4. 2022Cross-modal Memory Networks for Radiology Report Generationcited by 183
  5. 2024CheXpert Plus: Augmenting a Large Chest X-ray Dataset with Text Radiology Reports, Patient Demographics and Additional Image Formatscited by 100
  6. 2024CXPMRG-Bench: Pre-training and Benchmarking for X-ray Medical Report Generation on CheXpert Plus Datasetcited by 33
  7. 2024Image-aware Evaluation of Generated Medical Reportscited by 3
  8. 2024Designing a Robust Radiology Report Generation Systemcited by 1

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Adaptive Deep Learning for Breast Cancer Subtype Prediction Via Misprediction Risk Analysiscited by 0
  2. 2025Privacy-Preserving Chest X-ray Report Generation via Multimodal Federated Learning with ViT and GPT-2cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024CXR-Agent: Vision-language models for chest X-ray interpretation with uncertainty aware radiology reporting
  2. 2024Gla-AI4BioMed at RRG24: Visual Instruction-tuned Adaptation for Radiology Report Generation
  3. 2025Any-to-Any Vision-Language Model for Multimodal X-ray Imaging and Radiological Report Generation
  4. 2022RoentGen: Vision-Language Foundation Model for Chest X-ray Generation
  5. 2023MAIRA-1: A specialised large multimodal model for radiology report generation
  6. 2023XrayGPT: Chest Radiographs Summarization using Medical Vision-Language Models
  7. 2023MedXChat: A Unified Multimodal Large Language Model Framework towards CXRs Understanding and Generation
  8. 2026RadJEPA: Radiology Encoder for Chest X-Rays via Joint Embedding Predictive Architecture