mlsanityData to 7 Oct 2026

Improving Medical Visual Representations via Radiology Report Generation

arXiv 2310.19635 ↗First posted 30 Oct 2023cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Vision-language pretraining has been shown to produce high-quality visual encoders which transfer efficiently to downstream computer vision tasks. Contrastive learning approaches have increasingly been adopted for medical vision language pretraining (MVLP), yet recent developments in generative AI offer new modeling alternatives. This paper introduces RadTex, a CNN-encoder transformer-decoder architecture optimized for radiology. We explore bidirectional captioning as an alternative MVLP strategy and demonstrate that RadTex's captioning pretraining is competitive with established contrastive methods, achieving a CheXpert macro-AUC of 89.4%. Additionally, RadTex's lightweight text decoder not only generates clinically relevant radiology reports (macro-F1 score of 0.349), but also provides targeted, interactive responses, highlighting the utility of bidirectional captioning in advancing medical image analysis.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 46 of its 60 references that are in the map. First posted on arXiv 30 Oct 2023.

Builds on

46 of 60 references.

References · 46

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  9. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  11. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  12. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
34 more
  1. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  2. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,410
  3. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  4. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  5. 2017ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseasescited by 969
  6. 2023Large Language Modelscited by 774
  7. 2018Object Hallucination in Image Captioningcited by 661
  8. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
  9. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  10. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  11. 2020Generating Radiology Reports via Memory-driven Transformercited by 363
  12. 2019Meshed-Memory Transformer for Image Captioningcited by 343
  13. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
  14. 2019Lookahead Optimizer: k steps forward, 1 step backcited by 333
  15. 2020CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERTcited by 329
  16. 2023Towards Generalist Biomedical AIcited by 286
  17. 2016Using the Output Embedding to Improve Language Modelscited by 266
  18. 2017On the Automatic Generation of Medical Imaging Reportscited by 215
  19. 2023Learning to Exploit Temporal Structure for Biomedical Vision-Language Processingcited by 210
  20. 2022Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learningcited by 159
  21. 2018Hybrid Retrieval-Generation Reinforced Agent for Medical Image Report Generationcited by 141
  22. 2018TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-rayscited by 139
  23. 2020Improving Factual Completeness and Consistency of Image-to-Text Radiology Report Generationcited by 139
  24. 2021From Show to Tell: A Survey on Deep Learning-based Image Captioningcited by 137
  25. 2023Knowledge-enhanced Visual-Language Pre-training on Chest Radiology Imagescited by 128
  26. 2020Boosting Contrastive Self-Supervised Learning with False Negative Cancellationcited by 119
  27. 2022Improving Chest X-Ray Report Generation by Leveraging Warm Startingcited by 93
  28. 2023PRIOR: Prototype Representation Joint Learning from Medical Images and Reportscited by 55
  29. 2022Improving Radiology Report Generation Systems by Removing Hallucinated References to Non-existent Priorscited by 48
  30. 2020Joint Modeling of Chest Radiographs and Radiology Reports for Pulmonary Edema Assessmentcited by 37
  31. 2021Multimodal Representation Learning via Maximization of Local Mutual Informationcited by 27
  32. 2020Deep Learning to Quantify Pulmonary Edema in Chest Radiographscited by 10
  33. 2022COVIDx CXR-3: A Large-Scale, Open-Source Benchmark Dataset of Chest X-ray Images for Computer-Aided COVID-19 Diagnosticscited by 10
  34. 2023Improving Zero-Shot Detection of Low Prevalence Chest Pathologies using Domain Pre-trained Language Modelscited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026RadJEPA: Radiology Encoder for Chest X-Rays via Joint Embedding Predictive Architecture
  2. 2023Multi-modal Pre-training for Medical Vision-language Understanding and Generation: An Empirical Study with A New Benchmark
  3. 2026U-VLM: Hierarchical Vision Language Modeling for Report Generation
  4. 2025A Reality Check of Vision-Language Pre-training in Radiology: Have We Progressed Using Text?
  5. 2025More performant and scalable: Rethinking contrastive vision-language pre-training of radiology in the LLM era
  6. 2024RadCLIP: Enhancing Radiologic Image Analysis through Contrastive Language-Image Pre-training
  7. 2021Multi-modal Understanding and Generation for Medical Images and Text via Vision-Language Pre-Training
  8. 2024Advancing Medical Radiograph Representation Learning: A Hybrid Pre-training Paradigm with Multilevel Semantic Granularity