Vision Language Models versus Machine Learning Models Performance on Polyp Detection and Classification in Colonoscopy Images
On the map: Multimodal LLMs / vision-language models → Medical vision-language models
Abstract (arXiv)
Introduction: This study provides a comprehensive performance assessment of vision-language models (VLMs) against established convolutional neural networks (CNNs) and classic machine learning models (CMLs) for computer-aided detection (CADe) and computer-aided diagnosis (CADx) of colonoscopy polyp images. Method: We analyzed 2,258 colonoscopy images with corresponding pathology reports from 428 patients. We preprocessed all images using standardized techniques (resizing, normalization, and augmentation) and implemented a rigorous comparative framework evaluating 11 distinct models: ResNet50, 4 CMLs (random forest, support vector machine, logistic regression, decision tree), two specialized contrastive vision language encoders (CLIP, BiomedCLIP), and three general-purpose VLMs ( GPT-4 Gemini-1.5-Pro, Claude-3-Opus). Our performance assessment focused on two clinical tasks: polyp detection (CADe) and classification (CADx). Result: In polyp detection, ResNet50 achieved the best performance (F1: 91.35%, AUROC: 0.98), followed by BiomedCLIP (F1: 88.68%, AUROC: [AS1] ). GPT-4 demonstrated comparable effectiveness to traditional machine learning approaches (F1: 81.02%, AUROC: [AS2] ), outperforming other general-purpose VLMs. For polyp classification, performance rankings remained consistent but with lower overall metrics. ResNet50 maintained the highest efficacy (weighted F1: 74.94%), while GPT-4 demonstrated moderate capability (weighted F1: 41.18%), significantly exceeding other VLMs (Claude-3-Opus weighted F1: 25.54%, Gemini 1.5 Pro weighted F1: 6.17%). Conclusion: CNNs remain superior for both CADx and CADe tasks. However, VLMs like BioMedCLIP and GPT-4 may be useful for polyp detection tasks where training CNNs is not feasible.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 11 of its 44 references that are in the map. First posted on arXiv 27 Mar 2025.
Builds on
11 of 44 references.
References · 11
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2023GPT-4 Technical Reportcited by 18,396
- 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
- 2016Densely Connected Convolutional Networkscited by 6,720
- 2014Going Deeper with Convolutionscited by 4,241
- 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
- 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 520
- 2024ProSA: Assessing and Understanding the Prompt Sensitivity of LLMscited by 149
- 2024Hidden flaws behind expert-level accuracy of multimodal GPT-4 vision in medicinecited by 31
- 2023GPT-4 Vision on Medical Image Classification -- A Case Study on COVID-19 Datasetcited by 7
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Performance of large language models in the optical diagnosis of colorectal polyps
- 2024Frontiers in Intelligent Colonoscopy
- 2025LDP: Parameter-Efficient Fine-Tuning of Multimodal LLM for Medical Report Generation
- 2025Synthetic Data-Driven Multi-Architecture Framework for Automated Polyp Segmentation Through Integrated Detection and Mask Generation
- 2026A report-grounded vision-language foundation model for colonoscopy from 280000 routine reports
- 2025Diverse Image Generation with Diffusion Models and Cross Class Label Learning for Polyp Classification
- 2024Polyp-E: Benchmarking the Robustness of Deep Segmentation Models via Polyp Editing
- 2025Challenging Vision-Language Models with Surgical Data: A New Dataset and Broad Benchmarking Study