mlsanityData to 14 Oct 2026

Medical Image Captioning via Generative Pretrained Transformers

arXiv 2209.13983 ↗First posted 28 Sep 2022cs.CV cs.AIcited by 20

Not on the map. Why?

Abstract (arXiv)

The automatic clinical caption generation problem is referred to as proposed model combining the analysis of frontal chest X-Ray scans with structured patient information from the radiology records. We combine two language models, the Show-Attend-Tell and the GPT-3, to generate comprehensive and descriptive radiology records. The proposed combination of these models generates a textual summary with the essential information about pathologies found, their location, and the 2D heatmaps localizing each pathology on the original X-Ray scans. The proposed model is tested on two medical datasets, the Open-I, MIMIC-CXR, and the general-purpose MS-COCO. The results measured with the natural language assessment metrics prove their efficient applicability to the chest X-Ray image captioning.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Builds on 45 of its 73 references that are in the map. First posted on arXiv 28 Sep 2022.

Builds on

45 of 73 references.

References · 45

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  7. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  8. 2016Densely Connected Convolutional Networkscited by 6,720
  9. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  10. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  11. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  12. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
33 more
  1. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,423
  2. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,423
  3. 2017ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseasescited by 972
  4. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 968
  5. 2014Show and Tell: A Neural Image Caption Generatorcited by 879
  6. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 824
  7. 2014Long-term Recurrent Convolutional Networks for Visual Recognition and Descriptioncited by 484
  8. 2017CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learningcited by 461
  9. 2020Generating Radiology Reports via Memory-driven Transformercited by 368
  10. 2016Image Captioning with Semantic Attentioncited by 260
  11. 2017On the Automatic Generation of Medical Imaging Reportscited by 216
  12. 2021VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioningcited by 191
  13. 2020When Radiology Report Generation Meets Knowledge Graphcited by 145
  14. 2018TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-rayscited by 139
  15. 2019Clinically Accurate Chest X-Ray Report Generationcited by 139
  16. 2021TorchXRayVision: A library of chest X-ray datasets and modelscited by 131
  17. 2016Sequence-to-Sequence Learning as Beam-Search Optimizationcited by 127
  18. 2014From Captions to Visual Concepts and Backcited by 109
  19. 2017NegBio: a high-performance tool for negation and uncertainty detection in radiology reportscited by 79
  20. 2020Multi-domain Clinical Natural Language Processing with MedCAT: the Medical Concept Annotation Toolkitcited by 66
  21. 2020On the limits of cross-domain generalization in automated X-ray predictioncited by 62
  22. 2019Automatic Radiology Report Generation based on Multi-view Image Fusion and Medical Concept Enrichmentcited by 60
  23. 2016Learning to Read Chest X-Rays: Recurrent Neural Cascade Model for Automated Image Annotationcited by 59
  24. 2017MDNet: A Semantically and Visually Interpretable Medical Image Diagnosis Networkcited by 56
  25. 2021Active Learning for Sequence Tagging with Deep Pre-trained Models and Bayesian Uncertainty Estimatescited by 39
  26. 2019A Survey on Biomedical Image Captioningcited by 30
  27. 2020Anomaly Detection in Medical Imaging with Deep Perceptual Autoencoderscited by 30
  28. 2019Encoder-Agnostic Adaptation for Conditional Language Generationcited by 27
  29. 2020Chest X-ray Report Generation through Fine-Grained Label Learningcited by 26
  30. 2020Diagnosing COVID-19 Pneumonia from X-Ray and CT Images using Deep Learning and Transfer Learning Algorithmscited by 24
  31. 2020Learning Visual-Semantic Embeddings for Reporting Abnormal Findings on Chest X-rayscited by 18
  32. 2018Producing radiologist-quality reports for interpretable artificial intelligencecited by 11
  33. 2020Deep Negative Volume Segmentationcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 7; 2025: 7; 2026: 2.

20 papers.

Sort:

By area:

  1. 2024Automated Radiology Report Generation: A Review of Recent Advancescited by 43
  2. 2024Towards a Holistic Framework for Multimodal Large Language Models in Three-dimensional Brain CT Report Generationcited by 31
  3. 2024Kvasir-VQA: A Text-Image Pair GI Tract Datasetcited by 23
  4. 20233D-MIR: A Benchmark and Empirical Study on 3D Medical Image Retrieval in Radiologycited by 12
  5. 2024Large Language Models in Biomedical and Health Informatics: A Review with Bibliometric Analysiscited by 10
  6. 2025Hallucination-Aware Multimodal Benchmark for Gastrointestinal Image Analysis with Large Vision-Language Modelscited by 9
  7. 2023Generalizing Medical Image Representations via Quaternion Wavelet Networkscited by 6
  8. 2024A Survey of Deep Learning-based Radiology Report Generation Using Multimodal Datacited by 6
  9. 2023Zero-shot Learning with Minimum Instruction to Extract Social Determinants and Family History from Clinical Notes using GPT Modelcited by 5
  10. 2023Deciphering Diagnoses: How Large Language Models Explanations Influence Clinical Decision Makingcited by 2
  11. 2024Stability Analysis of ChatGPT-based Sentiment Analysis in AI Quality Assurancecited by 2
  12. 2024Pixels to Prose: Understanding the art of Image Captioningcited by 1
8 more
  1. 2025Random Direct Preference Optimization for Radiography Report Generationcited by 1
  2. 2025Fine-Tuning MedGemma for Clinical Captioning to Enhance Multimodal RAG over Malaysia CPGscited by 1
  3. 2025VICCA: Visual Interpretation and Comprehension of Chest X-ray Anomalies in Generated Report Without Human Feedbackcited by 0
  4. 2025A Deep Learning Approach for Augmenting Perceptional Understanding of Histopathology Imagescited by 0
  5. 2025Boundary-Aware Vision Transformer for Angiography Vascular Network Segmentationcited by 0
  6. 2025Transformers in Medicine: Improving Vision-Language Alignment for Medical Image Captioningcited by 0
  7. 2026Cross-Modal Clinical Knowledge Integration for Mammography Report Generationcited by 0
  8. 2026MSM-Mem: A Universal Medical Structured Multimodal Memory Framework for Medical AI Agentscited by 0