# RadVLM: A Multitask Conversational Vision-Language Model for Radiology

arXiv: https://arxiv.org/abs/2502.03333 · HTML: https://mlsanity.com/paper/2502.03333/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 61 of its 92 references that are in the map.
- First posted on arXiv 5 Feb 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)

## Builds on (most cited first, 30 of 61)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day](https://mlsanity.com/paper/2306.00890/) (2023)
- [VILA: On Pre-training for Visual Language Models](https://mlsanity.com/paper/2312.07533/) (2023)
- [Instruction Tuning with GPT-4](https://mlsanity.com/paper/2304.03277/) (2023)
- [MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs](https://mlsanity.com/paper/1901.07042/) (2019)
- [Ferret: Refer and Ground Anything Anywhere at Any Granularity](https://mlsanity.com/paper/2310.07704/) (2023)
- [LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset](https://mlsanity.com/paper/2309.11998/) (2023)
- [Making the Most of Text Semantics to Improve Biomedical Vision--Language Processing](https://mlsanity.com/paper/2204.09817/) (2022)
- [CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT](https://mlsanity.com/paper/2004.09167/) (2020)
- [What matters when building vision-language models?](https://mlsanity.com/paper/2405.02246/) (2024)
- [RadGraph: Extracting Clinical Entities and Relations from Radiology Reports](https://mlsanity.com/paper/2106.14463/) (2021)

## Built on by (most cited first, 25 of 25)

- [CXReasonBench: A Benchmark for Evaluating Structured Diagnostic Reasoning in Chest X-rays](https://mlsanity.com/paper/2505.18087/) (2025)
- [Agentic Systems in Radiology: Design, Applications, Evaluation, and Challenges](https://mlsanity.com/paper/2510.09404/) (2025)
- [Enhancing Radiology Report Generation and Visual Grounding using Reinforcement Learning](https://mlsanity.com/paper/2512.10691/) (2025)
- [Structure is Supervision: Multiview Masked Autoencoders for Radiology](https://mlsanity.com/paper/2511.22294/) (2025)
- [ProtoMedAgent: Multimodal Clinical Interpretability via Privacy-Aware Agentic Workflows](https://mlsanity.com/paper/2605.14113/) (2026)
- [The Path to Self-Evolving Clinical Systems: Scaling Medical Agents from Assistance to Autonomy](https://mlsanity.com/paper/2607.11175/) (2026)
- [Enhancing Abnormality Grounding for Vision Language Models with Knowledge Descriptions](https://mlsanity.com/paper/2503.03278/) (2025)
- [Knowledge to Sight: Reasoning over Visual Attributes via Knowledge Decomposition for Abnormality Grounding](https://mlsanity.com/paper/2508.04572/) (2025)
- [AnatomiX, an Anatomy-Aware Grounded Multimodal Large Language Model for Chest X-Ray Interpretation](https://mlsanity.com/paper/2601.03191/) (2026)
- [SemEnrich: Self-Supervised Semantic Enrichment of Radiology Reports for Vision-Language Learning](https://mlsanity.com/paper/2604.09887/) (2026)
- [Mind the Tool Failures: Achieving Synergistic Tool Gains for Medical Agents](https://mlsanity.com/paper/2605.26691/) (2026)
- [Hard Vision, Easy Vision: What GPT-6 Astra Reveals Across Computer Vision](https://mlsanity.com/paper/2609.35718/) (2026)
- [Interpreting Chest X-rays Like a Radiologist: A Benchmark with Clinical Reasoning](https://mlsanity.com/paper/2505.23143/) (2025)
- [Analysis of Blood Report Images Using General Purpose Vision-Language Models](https://mlsanity.com/paper/2509.06033/) (2025)
- [LoFi: Location-Aware Fine-Grained Representation Learning for Chest X-ray](https://mlsanity.com/paper/2603.19451/) (2026)
- [MedRoute: RL-Based Dynamic Specialist Routing in Multi-Agent Medical Diagnosis](https://mlsanity.com/paper/2604.06180/) (2026)
- [Dynamic Decision Learning: Test-Time Evolution for Abnormality Grounding in Rare Diseases](https://mlsanity.com/paper/2604.24972/) (2026)
- [Universal Boosts, Specific Suppressors: Sparse Autoencoder Steering of Medical Vision-Language Models](https://mlsanity.com/paper/2605.24977/) (2026)
- [SHOVIR: A Benchmark for Evaluating Vision Shortcut Learning in Radiology Report Generation](https://mlsanity.com/paper/2606.30201/) (2026)
- [Generalize LMMs to Versatile Visual Modalities via Fabricated Modality Synthesis](https://mlsanity.com/paper/2607.10308/) (2026)
- [KANEx: Translating Kolmogorov-Arnold Networks' Interpretability to Medical Explainability](https://mlsanity.com/paper/2607.24730/) (2026)
- [Location-Aware Fine-Grained Representation Learning for Medical Vision Foundation Models](https://mlsanity.com/paper/2608.00976/) (2026)
- [CheXGround: Anatomical Region Tokens for Grounded Longitudinal Chest X-ray Interpretation](https://mlsanity.com/paper/2608.30758/) (2026)
- [BrainDiff: Longitudinal Report Generation for Multimodal Brain MRI](https://mlsanity.com/paper/2609.00593/) (2026)
- [Reading the Whole Heart: Latent-Attention Masked Autoencoders for Multimodal Cardiac Representation Learning](https://mlsanity.com/paper/2609.12035/) (2026)

## Abstract (arXiv)

The widespread use of chest X-rays (CXRs), coupled with a shortage of radiologists, has driven growing interest in automated CXR analysis and AI-assisted reporting. While existing vision-language models (VLMs) show promise in specific tasks such as report generation or abnormality detection, they often lack support for interactive diagnostic capabilities. In this work we present RadVLM, a compact, multitask conversational foundation model designed for CXR interpretation. To this end, we curate a large-scale instruction dataset comprising over 1 million image-instruction pairs containing both single-turn tasks -- such as report generation, abnormality classification, and visual grounding -- and multi-turn, multi-task conversational interactions. After fine-tuning RadVLM on this instruction dataset, we evaluate it across different tasks along with re-implemented baseline VLMs. Our results show that RadVLM achieves state-of-the-art performance in conversational capabilities and visual grounding while remaining competitive in other radiology tasks. Ablation studies further highlight the benefit of joint training across multiple tasks, particularly for scenarios with limited annotated data. Together, these findings highlight the potential of RadVLM as a clinically relevant AI assistant, providing structured CXR interpretation and conversational capabilities to support more effective and accessible diagnostic workflows.