mlsanityData to 7 Oct 2026

RadVLM: A Multitask Conversational Vision-Language Model for Radiology

arXiv 2502.03333 ↗First posted 5 Feb 2025cs.CV cs.AIcited by 24

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

The widespread use of chest X-rays (CXRs), coupled with a shortage of radiologists, has driven growing interest in automated CXR analysis and AI-assisted reporting. While existing vision-language models (VLMs) show promise in specific tasks such as report generation or abnormality detection, they often lack support for interactive diagnostic capabilities. In this work we present RadVLM, a compact, multitask conversational foundation model designed for CXR interpretation. To this end, we curate a large-scale instruction dataset comprising over 1 million image-instruction pairs containing both single-turn tasks -- such as report generation, abnormality classification, and visual grounding -- and multi-turn, multi-task conversational interactions. After fine-tuning RadVLM on this instruction dataset, we evaluate it across different tasks along with re-implemented baseline VLMs. Our results show that RadVLM achieves state-of-the-art performance in conversational capabilities and visual grounding while remaining competitive in other radiology tasks. Ablation studies further highlight the benefit of joint training across multiple tasks, particularly for scenarios with limited annotated data. Together, these findings highlight the potential of RadVLM as a clinically relevant AI assistant, providing structured CXR interpretation and conversational capabilities to support more effective and accessible diagnostic workflows.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 61 of its 92 references that are in the map. First posted on arXiv 5 Feb 2025.

Builds on

61 of 92 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024MAIRA-Seg: Enhancing Radiology Report Generation with Segmentation-Aware Multimodal Large Language Models18 Nov
  2. 2024M4CXR: Exploring Multi-task Potentials of Multi-modal Large Language Models for Chest X-ray Interpretation29 Aug
  3. 2023MAIRA-1: A specialised large multimodal model for radiology report generation22 Nov
  4. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day1 Jun

Other references · 57

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2024The Llama 3 Herd of Modelscited by 14,192
  7. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  8. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  9. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  10. 2023Visual Instruction Tuningcited by 7,792
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  12. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
38 more
  1. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,829
  2. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  3. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  4. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  5. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  6. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  7. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  8. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,410
  9. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  10. 2023Instruction Tuning with GPT-4cited by 676
  11. 2019MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographscited by 590
  12. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  13. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 449
  14. 2022Making the Most of Text Semantics to Improve Biomedical Vision--Language Processingcited by 336
  15. 2020CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERTcited by 329
  16. 2024What matters when building vision-language models?cited by 321
  17. 2021RadGraph: Extracting Clinical Entities and Relations from Radiology Reportscited by 310
  18. 2019PadChest: A large chest x-ray image dataset with multi-label annotated reportscited by 308
  19. 2020VinDr-CXR: An open dataset of chest X-rays with radiologist's annotationscited by 236
  20. 2024Capabilities of Gemini Models in Medicinecited by 231
  21. 2024AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environmentscited by 215
  22. 2024Building and better understanding vision-language models: insights and future directionscited by 169
  23. 2024MAIRA-2: Grounded Radiology Report Generationcited by 140
  24. 2023Interactive and Explainable Region-guided Radiology Report Generationcited by 139
  25. 2024GREEN: Generative Radiology Report Evaluation and Error Notationcited by 124
  26. 2024CheXpert Plus: Augmenting a Large Chest X-ray Dataset with Text Radiology Reports, Patient Demographics and Additional Image Formatscited by 100
  27. 2024Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Modelscited by 97
  28. 2021Chest ImaGenome Dataset for Clinical Reasoningcited by 89
  29. 2024Advancing Multimodal Medical Capabilities of Geminicited by 88
  30. 2021Deep Learning for Chest X-ray Analysis: A Surveycited by 84
  31. 2022Improving the Factual Correctness of Radiology Report Generation with Semantic Rewardscited by 82
  32. 2024Towards Conversational Diagnostic AIcited by 79
  33. 2024A Vision-Language Foundation Model to Enhance Efficiency of Chest X-ray Interpretationcited by 79
  34. 2022Any-resolution Training for High-resolution Image Synthesiscited by 72
  35. 2021Radiology Report Generation with a Learned Knowledge Base and Multi-modal Alignmentcited by 71
  36. 2022MMDialog: A Large-scale Multi-turn Dialogue Dataset Towards Multi-modal Open-domain Conversationcited by 56
  37. 2021Progressive Transformer-Based Generation of Radiology Reportscited by 43
  38. 2024PadChest-GR: A Bilingual Chest X-ray Dataset for Grounded Radiology Report Generationcited by 35

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 16.

24 papers.

Sort:

By area:

  1. 2025CXReasonBench: A Benchmark for Evaluating Structured Diagnostic Reasoning in Chest X-rayscited by 5
  2. 2025Agentic Systems in Radiology: Design, Applications, Evaluation, and Challengescited by 5
  3. 2025Enhancing Radiology Report Generation and Visual Grounding using Reinforcement Learningcited by 4
  4. 2025Structure is Supervision: Multiview Masked Autoencoders for Radiologycited by 2
  5. 2026ProtoMedAgent: Multimodal Clinical Interpretability via Privacy-Aware Agentic Workflowscited by 2
  6. 2025Enhancing Abnormality Grounding for Vision Language Models with Knowledge Descriptionscited by 1
  7. 2025Knowledge to Sight: Reasoning over Visual Attributes via Knowledge Decomposition for Abnormality Groundingcited by 1
  8. 2026AnatomiX, an Anatomy-Aware Grounded Multimodal Large Language Model for Chest X-Ray Interpretationcited by 1
  9. 2026SemEnrich: Self-Supervised Semantic Enrichment of Radiology Reports for Vision-Language Learningcited by 1
  10. 2026Mind the Tool Failures: Achieving Synergistic Tool Gains for Medical Agentscited by 1
  11. 2026The Path to Self-Evolving Clinical Systems: Scaling Medical Agents from Assistance to Autonomycited by 1
  12. 2026Hard Vision, Easy Vision: What GPT-6 Astra Reveals Across Computer Visioncited by 1
12 more
  1. 2025Interpreting Chest X-rays Like a Radiologist: A Benchmark with Clinical Reasoningcited by 0
  2. 2025Analysis of Blood Report Images Using General Purpose Vision-Language Modelscited by 0
  3. 2026LoFi: Location-Aware Fine-Grained Representation Learning for Chest X-raycited by 0
  4. 2026MedRoute: RL-Based Dynamic Specialist Routing in Multi-Agent Medical Diagnosiscited by 0
  5. 2026Universal Boosts, Specific Suppressors: Sparse Autoencoder Steering of Medical Vision-Language Modelscited by 0
  6. 2026SHOVIR: A Benchmark for Evaluating Vision Shortcut Learning in Radiology Report Generationcited by 0
  7. 2026Generalize LMMs to Versatile Visual Modalities via Fabricated Modality Synthesiscited by 0
  8. 2026KANEx: Translating Kolmogorov-Arnold Networks' Interpretability to Medical Explainabilitycited by 0
  9. 2026Location-Aware Fine-Grained Representation Learning for Medical Vision Foundation Modelscited by 0
  10. 2026CheXGround: Anatomical Region Tokens for Grounded Longitudinal Chest X-ray Interpretationcited by 0
  11. 2026BrainDiff: Longitudinal Report Generation for Multimodal Brain MRIcited by 0
  12. 2026Reading the Whole Heart: Latent-Attention Masked Autoencoders for Multimodal Cardiac Representation Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023RaDialog: A Large Vision-Language Model for Radiology Report Generation and Conversational Assistance
  2. 2026SentZero: An Enhanced Sentence-Centric Vision-Language Pretraining for Multi-Task Zero-Shot Chest X-Ray Analysis
  3. 2023MedXChat: A Unified Multimodal Large Language Model Framework towards CXRs Understanding and Generation
  4. 2024CXR-Agent: Vision-language models for chest X-ray interpretation with uncertainty aware radiology reporting
  5. 2023XrayGPT: Chest Radiographs Summarization using Medical Vision-Language Models
  6. 2026Harrison.Rad 1.5 Technical Report: A radiology foundation model that can draft reports from images, priors and clinical context
  7. 2025A Generative Framework for Bidirectional Image-Report Understanding in Chest Radiography
  8. 2024Multimodal Healthcare AI: Identifying and Designing Clinically Relevant Vision-Language Applications for Radiology