MiniGPT-Pancreas: Multimodal Large Language Model for Pancreas Cancer Classification and Detection
On the map: Multimodal LLMs / vision-language models → Medical vision-language models
Abstract (arXiv)
Problem: Pancreas radiological imaging is challenging due to the small size, blurred boundaries, and variability of shape and position of the organ among patients. Goal: In this work we present MiniGPT-Pancreas, a Multimodal Large Language Model (MLLM), as an interactive chatbot to support clinicians in pancreas cancer diagnosis by integrating visual and textual information. Methods: MiniGPT-v2, a general-purpose MLLM, was fine-tuned in a cascaded way for pancreas detection, tumor classification, and tumor detection with multimodal prompts combining questions and computed tomography scans from the National Institute of Health (NIH), and Medical Segmentation Decathlon (MSD) datasets. The AbdomenCT-1k dataset was used to detect the liver, spleen, kidney, and pancreas. Results: MiniGPT-Pancreas achieved an Intersection over Union (IoU) of 0.595 and 0.550 for the detection of pancreas on NIH and MSD datasets, respectively. For the pancreas cancer classification task on the MSD dataset, accuracy, precision, and recall were 0.876, 0.874, and 0.878, respectively. When evaluating MiniGPT-Pancreas on the AbdomenCT-1k dataset for multi-organ detection, the IoU was 0.8399 for the liver, 0.722 for the kidney, 0.705 for the spleen, and 0.497 for the pancreas. For the pancreas tumor detection task, the IoU score was 0.168 on the MSD dataset. Conclusions: MiniGPT-Pancreas represents a promising solution to support clinicians in the classification of pancreas images with pancreas tumors. Future research is needed to improve the score on the detection task, especially for pancreas tumors.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 30 of its 55 references that are in the map. First posted on arXiv 20 Dec 2024.
Builds on
30 of 55 references.
Closest ancestors · 2
Same direction, newest first
- 2023XrayGPT: Chest Radiographs Summarization using Medical Vision-Language Models13 Jun
- 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day1 Jun
Other references · 28
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
- 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
- 2023VILA: On Pre-training for Visual Language Modelscited by 686
- 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
- 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
- 2019MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographscited by 590
- 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 564
- 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
- 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 506
- 2019The Liver Tumor Segmentation Benchmark (LiTS)cited by 504
16 more
- 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 467
- 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 387
- 2019A large annotated medical image dataset for the development and evaluation of segmentation algorithmscited by 371
- 2023Segment Anything Model for Medical Image Analysis: an Experimental Studycited by 291
- 2023Towards Generalist Biomedical AIcited by 286
- 2020AbdomenCT-1K: Is Abdominal Organ Segmentation A Solved Problem?cited by 222
- 2023PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documentscited by 212
- 2023Learning to Exploit Temporal Structure for Biomedical Vision-Language Processingcited by 210
- 2023Towards Generalist Foundation Model for Radiology by Leveraging Web-scale 2D&3D Medical Datacited by 176
- 2015DeepOrgan: Multi-level Deep Convolutional Networks for Automated Pancreas Segmentationcited by 158
- 2022Multi-Modal Masked Autoencoders for Medical Vision-and-Language Pre-Trainingcited by 104
- 2021Medical Visual Question Answering: A Surveycited by 93
- 2024ROCOv2: Radiology Objects in COntext Version 2, an Updated Multimodal Image Datasetcited by 54
- 2024MiniGPT-Med: Large Language Model as a General Interface for Radiology Diagnosiscited by 28
- 2019Deep Q Learning Driven CT Pancreas Segmentation with Geometry-Aware U-Netcited by 15
- 2024Deep Learning for Pancreas Segmentation: a Systematic Reviewcited by 4
Built on by
Citing papers per year: 2025: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026DB-MSMUNet:Dual Branch Multi-scale Mamba UNet for Pancreatic CT Scans Segmentation
- 2025Leveraging Fine-Tuned Large Language Models for Interpretable Pancreatic Cystic Lesion Feature Extraction and Risk Categorization
- 2024Potential of Multimodal Large Language Models for Data Mining of Medical Images and Free-text Reports
- 2025LLaVA-RadZ: Can Multimodal Large Language Models Effectively Tackle Zero-shot Radiology Recognition?
- 2024On Large Visual Language Models for Medical Imaging Analysis: An Empirical Study
- 2024Pancreatic Tumor Segmentation as Anomaly Detection in CT Images Using Denoising Diffusion Models
- 2024An Early Investigation into the Utility of Multimodal Large Language Models in Medical Imaging
- 2024A Refer-and-Ground Multimodal Large Language Model for Biomedicine