mlsanityData to 7 Oct 2026

XrayGPT: Chest Radiographs Summarization using Medical Vision-Language Models

arXiv 2306.07971 ↗First posted 13 Jun 2023cs.CVcited by 61

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

The latest breakthroughs in large vision-language models, such as Bard and GPT-4, have showcased extraordinary abilities in performing a wide range of tasks. Such models are trained on massive datasets comprising billions of public image-text pairs with diverse tasks. However, their performance on task-specific domains, such as radiology, is still under-investigated and potentially limited due to a lack of sophistication in understanding biomedical images. On the other hand, conversational medical models have exhibited remarkable success but have mainly focused on text-based analysis. In this paper, we introduce XrayGPT, a novel conversational medical vision-language model that can analyze and answer open-ended questions about chest radiographs. Specifically, we align both medical visual encoder (MedClip) with a fine-tuned large language model (Vicuna), using a simple linear transformation. This alignment enables our model to possess exceptional visual conversation abilities, grounded in a deep understanding of radiographs and medical domain knowledge. To enhance the performance of LLMs in the medical context, we generate ~217k interactive and high-quality summaries from free-text radiology reports. These summaries serve to enhance the performance of LLMs through the fine-tuning process. Our approach opens up new avenues the research for advancing the automated analysis of chest radiographs. Our open-source demos, models, and instruction sets are available at: https://github.com/mbzuai-oryx/XrayGPT.

Key facts. Cited by 61 papers in the map (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 14 of its 23 references that are in the map. First posted on arXiv 13 Jun 2023.

Builds on

14 of 23 references.

References · 14

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  3. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  4. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  5. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  6. 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 564
  7. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  8. 2023ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledgecited by 324
  9. 2023MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Datacited by 282
  10. 2016SummaRuNNer: A Recurrent Neural Network based Sequence Model for Extractive Summarization of Documentscited by 245
  11. 2018A Comprehensive Survey of Deep Learning for Image Captioningcited by 213
  12. 2023DoctorGLM: Fine-tuning your Chinese Doctor is not a Herculean Taskcited by 130
2 more
  1. 2016Neural Summarization by Extracting Sentences and Wordscited by 114
  2. 2023Q2ATransformer: Improving Medical VQA via an Answer Querying Decodercited by 31

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 10; 2024: 19; 2025: 21; 2026: 11.

61 papers.

Sort:

By area:

  1. 2023GeoChat: Grounded Large Vision-Language Model for Remote Sensingcited by 320
  2. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
  3. 2024Advancing Multimodal Medical Capabilities of Geminicited by 88
  4. 2024Foundation Model for Advancing Healthcare: Challenges, Opportunities, and Future Directionscited by 77
  5. 2023Foundational Models in Medical Imaging: A Comprehensive Survey and Future Visioncited by 74
  6. 2024A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicinecited by 45
  7. 2025EndoChat: Grounded Multimodal Large Language Model for Endoscopic Surgerycited by 37
  8. 2024Dia-LLaMA: Towards Large Language Model-driven CT Report Generationcited by 32
  9. 2024AgroGPT: Efficient Agricultural Vision-Language Model with Expert Tuningcited by 23
  10. 2024EVA-X: A Foundation Model for General Chest X-ray Analysis with Self-supervised Learningcited by 22
  11. 2024MedSumm: A Multimodal Approach to Summarizing Code-Mixed Hindi-English Clinical Queriescited by 19
  12. 2024A Survey on Trustworthiness in Foundation Models for Medical Image Analysiscited by 16
38 more
  1. 2025Multimodal Large Language Models for Medicine: A Comprehensive Surveycited by 16
  2. 2023A Systematic Review of Deep Learning-based Research on Radiology Report Generationcited by 14
  3. 2024CDChat: A Large Multimodal Model for Remote Sensing Change Descriptioncited by 13
  4. 2023Path to Medical AGI: Unify Domain-specific Medical LLMs with the Lowest Costcited by 12
  5. 2023Pragmatic Radiology Report Generationcited by 12
  6. 2025UniMedVL: Unifying Medical Multimodal Understanding and Generation through Observation-Knowledge-Analysiscited by 12
  7. 2024Seeing is Believing: Mitigating Hallucination in Large Vision-Language Models via CLIP-Guided Decodingcited by 8
  8. 2025MoMA: A Mixture-of-Multimodal-Agents Architecture for Enhancing Clinical Prediction Modellingcited by 7
  9. 2025Toward a Vision-Language Foundation Model for Medical Data: Multimodal Dataset and Benchmarks for Vietnamese PET/CT Report Generationcited by 7
  10. 2023Multimodal Foundation Models For Echocardiogram Interpretationcited by 6
  11. 2023MedXChat: A Unified Multimodal Large Language Model Framework towards CXRs Understanding and Generationcited by 5
  12. 2023Breast Ultrasound Report Generation using LangChaincited by 5
  13. 2024Prompting Medical Large Vision-Language Models to Diagnose Pathologies by Visual Question Answeringcited by 5
  14. 2023CephGPT-4: An Interactive Multimodal Cephalometric Measurement and Diagnostic System with Visual Large Language Modelcited by 4
  15. 2024LiteGPT: Large Vision-Language Model for Joint Chest X-ray Localization and Classification Taskcited by 4
  16. 2025Zeus: Zero-shot LLM Instruction for Union Segmentation in Multimodal Medical Imagingcited by 4
  17. 2025LLM-driven Medical Report Generation via Communication-efficient Heterogeneous Federated Learningcited by 4
  18. 2025Skin-R1: Clinical Knowledge-Guided Dermatological Diagnosis Using Vision-Language Modelscited by 4
  19. 2024Evaluating Automated Radiology Report Quality through Fine-Grained Phrasal Grounding of Clinical Findingscited by 3
  20. 2025VinDr-CXR-VQA: A Visual Question Answering Dataset for Explainable Chest X-Ray Analysis with Multi-Task Learningcited by 3
  21. 2026ECHO: Efficient Chest X-ray Report Generation with One-step Block Diffusioncited by 3
  22. 2024Machine Learning and Transformers for Thyroid Carcinoma Diagnosis: A Reviewcited by 2
  23. 2024WoLF: Wide-scope Large Language Model Framework for CXR Understandingcited by 2
  24. 2024Inquire, Interact, and Integrate: A Proactive Agent Collaborative Framework for Zero-Shot Multimodal Medical Reasoningcited by 2
  25. 2024Boosting Medical Image-based Cancer Detection via Text-guided Supervision from Reportscited by 2
  26. 2025Recent Advances in Medical Image Classificationcited by 2
  27. 2025Recurrent Visual Feature Extraction and Stereo Attentions for CT Report Generationcited by 2
  28. 2025Benchmarking the Thinking Mode of Multimodal Large Language Models in Clinical Taskscited by 2
  29. 2024UlcerGPT: A Multimodal Approach Leveraging Large Language and Vision Models for Diabetic Foot Ulcer Image Transcriptioncited by 1
  30. 2024MiniGPT-Pancreas: Multimodal Large Language Model for Pancreas Cancer Classification and Detectioncited by 1
  31. 2025Enhancing Multi-task Learning Capability of Medical Generalist Foundation Model via Image-centric Multi-annotation Datacited by 1
  32. 2025ChestGPT: Integrating Large Language Models and Vision Transformers for Disease Detection and Localization in Chest X-Rayscited by 1
  33. 2025A Multi-Agent System for Complex Reasoning in Radiology Visual Question Answeringcited by 1
  34. 2025Revolutionizing Precise Low Back Pain Diagnosis via Contrastive Learningcited by 1
  35. 2025Analysing Environmental Efficiency in AI for X-Ray Diagnosiscited by 1
  36. 2025Why Text Prevails: Vision May Undermine Multimodal Medical Decision Makingcited by 1
  37. 2026From Attribution to Abstention: Training-Free Attention-Based Auditing for Clinical Summarizationcited by 1
  38. 2026Multi-Modal Multi-Agent Reinforcement Learning for Radiology Report Generationcited by 1

50 most cited of 61 All 61 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025RadVLM: A Multitask Conversational Vision-Language Model for Radiology
  2. 2026SentZero: An Enhanced Sentence-Centric Vision-Language Pretraining for Multi-Task Zero-Shot Chest X-Ray Analysis
  3. 2025XBench: A Comprehensive Benchmark for Visual-Language Explanations in Chest Radiography
  4. 2025A Reality Check of Vision-Language Pre-training in Radiology: Have We Progressed Using Text?
  5. 2023RaDialog: A Large Vision-Language Model for Radiology Report Generation and Conversational Assistance
  6. 2025A Generative Framework for Bidirectional Image-Report Understanding in Chest Radiography
  7. 2024MiniGPT-Med: Large Language Model as a General Interface for Radiology Diagnosis
  8. 2025Adapting Lightweight Vision Language Models for Radiological Visual Question Answering