# MiniGPT-Med: Large Language Model as a General Interface for Radiology Diagnosis

arXiv: https://arxiv.org/abs/2407.04106 · HTML: https://mlsanity.com/paper/2407.04106/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 32 references that are in the map.
- First posted on arXiv 4 Jul 2024.

## Builds on (most cited first, 16 of 16)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day](https://mlsanity.com/paper/2306.00890/) (2023)
- [EVA-CLIP: Improved Training Techniques for CLIP at Scale](https://mlsanity.com/paper/2303.15389/) (2023)
- [BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining](https://mlsanity.com/paper/2210.10341/) (2022)
- [Med-Flamingo: a Multimodal Medical Few-shot Learner](https://mlsanity.com/paper/2307.15189/) (2023)
- [Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease prediction](https://mlsanity.com/paper/2005.12833/) (2020)
- [Learning to Exploit Temporal Structure for Biomedical Vision-Language Processing](https://mlsanity.com/paper/2301.04558/) (2023)
- [Towards Generalist Foundation Model for Radiology by Leveraging Web-scale 2D&3D Medical Data](https://mlsanity.com/paper/2308.02463/) (2023)
- [A Vision-Language Foundation Model to Enhance Efficiency of Chest X-ray Interpretation](https://mlsanity.com/paper/2401.12208/) (2024)
- [BERTHop: An Effective Vision-and-Language Model for Chest X-ray Disease Diagnosis](https://mlsanity.com/paper/2108.04938/) (2021)
- [Visual Question Answering in the Medical Domain](https://mlsanity.com/paper/2309.11080/) (2023)

## Built on by (most cited first, 28 of 28)

- [MMed-RAG: Versatile Multimodal RAG System for Medical Vision Language Models](https://mlsanity.com/paper/2410.13085/) (2024)
- [GEMeX: A Large-Scale, Groundable, and Explainable Medical VQA Benchmark for Chest X-ray Diagnosis](https://mlsanity.com/paper/2411.16778/) (2024)
- [Large Language Model with Region-guided Referring and Grounding for CT Report Generation](https://mlsanity.com/paper/2411.15539/) (2024)
- [Interpretable Bilingual Multimodal Large Language Model for Diverse Biomedical Tasks](https://mlsanity.com/paper/2410.18387/) (2024)
- [BiMediX2: Bio-Medical EXpert LMM for Diverse Medical Modalities](https://mlsanity.com/paper/2412.07769/) (2024)
- [E3D-GPT: Enhanced 3D Visual Foundation for Medical Vision-Language Model](https://mlsanity.com/paper/2410.14200/) (2024)
- [Multimodal Large Language Models for Medicine: A Comprehensive Survey](https://mlsanity.com/paper/2504.21051/) (2025)
- [MM-Skin: Enhancing Dermatology Vision-Language Model with an Image-Text Dataset Derived from Textbooks](https://mlsanity.com/paper/2505.06152/) (2025)
- [Anyprefer: An Agentic Framework for Preference Data Synthesis](https://mlsanity.com/paper/2504.19276/) (2025)
- [U2-BENCH: Benchmarking Large Vision-Language Models on Ultrasound Understanding](https://mlsanity.com/paper/2505.17779/) (2025)
- [MMXU: A Multi-Modal and Multi-X-ray Understanding Dataset for Disease Progression](https://mlsanity.com/paper/2502.11651/) (2025)
- [Taming Vision-Language Models for Medical Image Analysis: A Comprehensive Review](https://mlsanity.com/paper/2506.18378/) (2025)
- [Patient-Level Anatomy Meets Scanning-Level Physics: Personalized Federated Low-Dose CT Denoising Empowered by Large Language Model](https://mlsanity.com/paper/2503.00908/) (2025)
- [STING-BEE: Towards Vision-Language Model for Real-World X-ray Baggage Security Inspection](https://mlsanity.com/paper/2504.02823/) (2025)
- [SilVar-Med: A Speech-Driven Visual Language Model for Explainable Abnormality Detection in Medical Imaging](https://mlsanity.com/paper/2504.10642/) (2025)
- [OrthoDoc: Multimodal Large Language Model for Assisting Diagnosis in Computed Tomography](https://mlsanity.com/paper/2409.09052/) (2024)
- [MCA-RG: Enhancing LLMs with Medical Concept Alignment for Radiology Report Generation](https://mlsanity.com/paper/2507.06992/) (2025)
- [Clinical Cognition Alignment for Gastrointestinal Diagnosis with Multimodal LLMs](https://mlsanity.com/paper/2603.20698/) (2026)
- [Enhancing Reinforcement Learning for Radiology Report Generation with Evidence-aware Rewards and Self-correcting Preference Learning](https://mlsanity.com/paper/2604.13598/) (2026)
- [Enhancing Multi-task Learning Capability of Medical Generalist Foundation Model via Image-centric Multi-annotation Data](https://mlsanity.com/paper/2504.09967/) (2025)
- [Online Iterative Self-Alignment for Radiology Report Generation](https://mlsanity.com/paper/2505.11983/) (2025)
- [V-Loop: Visual Logical Loop Verification for Hallucination Detection in Medical Visual Question Answering](https://mlsanity.com/paper/2601.18240/) (2026)
- [Beyond a Single Frame: Multi-Frame Spatially Grounded Reasoning Across Volumetric MRI](https://mlsanity.com/paper/2604.15808/) (2026)
- [MiniGPT-Pancreas: Multimodal Large Language Model for Pancreas Cancer Classification and Detection](https://mlsanity.com/paper/2412.15925/) (2024)
- [ChestGPT: Integrating Large Language Models and Vision Transformers for Disease Detection and Localization in Chest X-Rays](https://mlsanity.com/paper/2507.03739/) (2025)
- [CogBench: A Large Language Model Benchmark for Multilingual Speech-Based Cognitive Impairment Assessment](https://mlsanity.com/paper/2508.03360/) (2025)
- [Knowledge to Sight: Reasoning over Visual Attributes via Knowledge Decomposition for Abnormality Grounding](https://mlsanity.com/paper/2508.04572/) (2025)
- [Interpreting Chest X-rays Like a Radiologist: A Benchmark with Clinical Reasoning](https://mlsanity.com/paper/2505.23143/) (2025)

## Abstract (arXiv)

Recent advancements in artificial intelligence (AI) have precipitated significant breakthroughs in healthcare, particularly in refining diagnostic procedures. However, previous studies have often been constrained to limited functionalities. This study introduces MiniGPT-Med, a vision-language model derived from large-scale language models and tailored for medical applications. MiniGPT-Med demonstrates remarkable versatility across various imaging modalities, including X-rays, CT scans, and MRIs, enhancing its utility. The model is capable of performing tasks such as medical report generation, visual question answering (VQA), and disease identification within medical imagery. Its integrated processing of both image and textual clinical data markedly improves diagnostic accuracy. Our empirical assessments confirm MiniGPT-Med's superior performance in disease grounding, medical report generation, and VQA benchmarks, representing a significant step towards reducing the gap in assisting radiology practice. Furthermore, it achieves state-of-the-art performance on medical report generation, higher than the previous best model by 19% accuracy. MiniGPT-Med promises to become a general interface for radiology diagnoses, enhancing diagnostic efficiency across a wide range of medical imaging applications.