mlsanityData to 7 Oct 2026

RaDialog: A Large Vision-Language Model for Radiology Report Generation and Conversational Assistance

arXiv 2311.18681 ↗First posted 30 Nov 2023cs.CV cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Conversational AI tools that can generate and discuss clinically correct radiology reports for a given medical image have the potential to transform radiology. Such a human-in-the-loop radiology assistant could facilitate a collaborative diagnostic process, thus saving time and improving the quality of reports. Towards this goal, we introduce RaDialog, the first thoroughly evaluated and publicly available large vision-language model for radiology report generation and interactive dialog. RaDialog effectively integrates visual image features and structured pathology findings with a large language model (LLM) while simultaneously adapting it to a specialized domain using parameter-efficient fine-tuning. To keep the conversational abilities of the underlying LLM, we propose a comprehensive, semi-automatically labeled, image-grounded instruct dataset for chest X-ray radiology tasks. By training with this dataset, our method achieves state-of-the-art clinical correctness in report generation and shows impressive abilities in interactive tasks such as correcting reports and answering questions, serving as a foundational step toward clinical dialog systems. Our code is available on github: https://github.com/ChantalMP/RaDialog.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 31 of its 46 references that are in the map. First posted on arXiv 30 Nov 2023.

Builds on

31 of 46 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023MAIRA-1: A specialised large multimodal model for radiology report generation22 Nov
  2. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day1 Jun

Other references · 29

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  8. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  9. 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 387
  10. 2020Generating Radiology Reports via Memory-driven Transformercited by 363
  11. 2020CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERTcited by 329
  12. 2023ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledgecited by 324
17 more
  1. 2023Towards Generalist Biomedical AIcited by 286
  2. 2023Learning to Exploit Temporal Structure for Biomedical Vision-Language Processingcited by 210
  3. 2023Towards Generalist Foundation Model for Radiology by Leveraging Web-scale 2D&3D Medical Datacited by 176
  4. 2020Improving Factual Completeness and Consistency of Image-to-Text Radiology Report Generationcited by 139
  5. 2023Dynamic Graph Enhanced Contrastive Learning for Chest X-ray Report Generationcited by 114
  6. 2023METransformer: Radiology Report Generation by Transformer with Multiple Learnable Expert Tokenscited by 98
  7. 2021Radiology Report Generation with a Learned Knowledge Base and Multi-modal Alignmentcited by 71
  8. 2021Knowledge Matters: Radiology Report Generation with General and Specific Knowledgecited by 68
  9. 2023KiUT: Knowledge-injected U-Transformer for Radiology Report Generationcited by 60
  10. 2021Progressive Transformer-Based Generation of Radiology Reportscited by 43
  11. 2021Weakly Supervised Contrastive Learning for Chest X-Ray Report Generationcited by 39
  12. 2023ORGAN: Observation-Guided Radiology Report Generation via Tree Reasoningcited by 39
  13. 2023Multimodal Image-Text Matching Improves Retrieval-based Chest X-Ray Report Generationcited by 37
  14. 2022Explaining Chest X-ray Pathologies in Natural Languagecited by 27
  15. 2023Complex Organ Mask Guided Radiology Report Generationcited by 22
  16. 2023Rad-ReStruct: A Novel VQA Benchmark and Method for Structured Radiology Reportingcited by 17
  17. 2022FlexR: Few-shot Classification with Language Embeddings for Structured Reporting of Chest X-rayscited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025RadVLM: A Multitask Conversational Vision-Language Model for Radiology
  2. 2024Multimodal Healthcare AI: Identifying and Designing Clinically Relevant Vision-Language Applications for Radiology
  3. 2024Multimodal Large Language Model driven Radiology Report Generation with Clinical Knowledge Enhancement
  4. 2025Visual Alignment of Medical Vision-Language Models for Grounded Radiology Report Generation
  5. 2024MAIRA-Seg: Enhancing Radiology Report Generation with Segmentation-Aware Multimodal Large Language Models
  6. 2023XrayGPT: Chest Radiographs Summarization using Medical Vision-Language Models
  7. 2024SERPENT-VLM : Self-Refining Radiology Report Generation Using Vision Language Models
  8. 2024Gla-AI4BioMed at RRG24: Visual Instruction-tuned Adaptation for Radiology Report Generation