mlsanityData to 14 Oct 2026

RepsNet: Combining Vision with Language for Automated Medical Reports

arXiv 2209.13171 ↗First posted 27 Sep 2022cs.CVcited by 14

Not on the map. Why?

Abstract (arXiv)

Writing reports by analyzing medical images is error-prone for inexperienced practitioners and time consuming for experienced ones. In this work, we present RepsNet that adapts pre-trained vision and language models to interpret medical images and generate automated reports in natural language. RepsNet consists of an encoder-decoder model: the encoder aligns the images with natural language descriptions via contrastive learning, while the decoder predicts answers by conditioning on encoded images and prior context of descriptions retrieved by nearest neighbor search. We formulate the problem in a visual question answering setting to handle both categorical and descriptive natural language answers. We perform experiments on two challenging tasks of medical visual question answering (VQA-Rad) and report generation (IU-Xray) on radiology image datasets. Results show that RepsNet outperforms state-of-the-art methods with 81.08 % classification accuracy on VQA-Rad 2018 and 0.58 BLEU-1 score on IU-Xray. Supplementary details are available at https://sites.google.com/view/repsnet

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 32 of its 45 references that are in the map. First posted on arXiv 27 Sep 2022.

Builds on

32 of 45 references.

References · 32

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  4. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,546
  5. 2015VQA: Visual Question Answeringcited by 2,506
  6. 2017Billion-scale similarity search with GPUscited by 2,497
  7. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  8. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  9. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,423
  10. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 703
  11. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 670
  12. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 641
20 more
  1. 2020Generating Radiology Reports via Memory-driven Transformercited by 368
  2. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 339
  3. 2020Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformerscited by 288
  4. 2018Bilinear Attention Networkscited by 280
  5. 2017On the Automatic Generation of Medical Imaging Reportscited by 216
  6. 2021Exploring and Distilling Posterior and Prior Knowledge for Radiology Report Generationcited by 162
  7. 2020When Radiology Report Generation Meets Knowledge Graphcited by 145
  8. 2018Hybrid Retrieval-Generation Reinforced Agent for Medical Image Report Generationcited by 141
  9. 2019Clinically Accurate Chest X-Ray Report Generationcited by 139
  10. 2020Learning Visual Representations with Caption Annotationscited by 123
  11. 2020A Survey of Knowledge-Enhanced Text Generationcited by 120
  12. 2020Contrastive Learning for Weakly Supervised Phrase Groundingcited by 102
  13. 2019Overcoming Data Limitation in Medical Visual Question Answeringcited by 71
  14. 2019Automatic Radiology Report Generation based on Multi-view Image Fusion and Medical Concept Enrichmentcited by 60
  15. 2020Show, Describe and Conclude: On Exploiting the Structure Information of Chest X-Ray Reportscited by 58
  16. 2021Multiple Meta-model Quantifying for Medical Visual Question Answeringcited by 54
  17. 2020XGPT: Cross-modal Generative Pre-Training for Image Captioningcited by 51
  18. 2019Learning Video Representations using Contrastive Bidirectional Transformercited by 39
  19. 2019Encoder-Agnostic Adaptation for Conditional Language Generationcited by 27
  20. 2021Variational Topic Inference for Chest X-Ray Report Generationcited by 16

Built on by

Cited by year

Citing papers per year: 2023: 5; 2024: 5; 2025: 4.

14 papers.

Sort:

By area:

  1. 2024Vision-Language Models for Medical Report Generation and Visual Question Answering: A Reviewcited by 125
  2. 2024Automated Radiology Report Generation: A Review of Recent Advancescited by 43
  3. 2023Complex Organ Mask Guided Radiology Report Generationcited by 22
  4. 2023Rad-ReStruct: A Novel VQA Benchmark and Method for Structured Radiology Reportingcited by 18
  5. 2023A Systematic Review of Deep Learning-based Research on Radiology Report Generationcited by 14
  6. 2025MM-Skin: Enhancing Dermatology Vision-Language Model with an Image-Text Dataset Derived from Textbookscited by 14
  7. 2023A scoping review on multimodal deep learning in biomedical images and textscited by 13
  8. 2025Taming Vision-Language Models for Medical Image Analysis: A Comprehensive Reviewcited by 9
  9. 2024Dragonfly: Multi-Resolution Zoom-In Encoding Enhances Vision-Language Modelscited by 8
  10. 2024A Survey of Deep Learning-based Radiology Report Generation Using Multimodal Datacited by 6
  11. 2024ORID: Organ-Regional Information Driven Framework for Radiology Report Generationcited by 3
  12. 2025Self-Supervised Anatomical Consistency Learning for Vision-Grounded Medical Report Generationcited by 3
2 more
  1. 2023Multimodal Prompt Retrieval for Generative Visual Question Answeringcited by 2
  2. 2025One Patient, Many Contexts: Scaling Medical AI with Contextual Intelligencecited by 2