# Can Large Language Models Challenge CNNs in Medical Image Analysis?

arXiv: https://arxiv.org/abs/2505.23503 · HTML: https://mlsanity.com/paper/2505.23503/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 27 references that are in the map.
- First posted on arXiv 29 May 2025.

## Builds on (most cited first, 14 of 14)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [UNETR: Transformers for 3D Medical Image Segmentation](https://mlsanity.com/paper/2103.10504/) (2021)
- [Explainability for Large Language Models: A Survey](https://mlsanity.com/paper/2309.01029/) (2023)
- [Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Model](https://mlsanity.com/paper/2211.02001/) (2022)
- [A review of deep learning in medical imaging: Imaging traits, technology trends, case studies with progress highlights, and future promises](https://mlsanity.com/paper/2008.09104/) (2020)
- [Can AI help in screening Viral and COVID-19 pneumonia?](https://mlsanity.com/paper/2003.13145/) (2020)
- [Unsupervised Medical Image Translation with Adversarial Diffusion Models](https://mlsanity.com/paper/2207.08208/) (2022)
- [Large language models in healthcare and medical domain: A review](https://mlsanity.com/paper/2401.06775/) (2023)
- [MMBERT: Multimodal BERT Pretraining for Improved Medical VQA](https://mlsanity.com/paper/2104.01394/) (2021)
- [Blockchain-Federated-Learning and Deep Learning Models for COVID-19 detection using CT Imaging](https://mlsanity.com/paper/2007.06537/) (2020)
- [ChatCAD+: Towards a Universal and Reliable Interactive CAD using LLMs](https://mlsanity.com/paper/2305.15964/) (2023)
- [On Large Visual Language Models for Medical Imaging Analysis: An Empirical Study](https://mlsanity.com/paper/2402.14162/) (2024)
- [Multi-task Paired Masking with Alignment Modeling for Medical Vision-Language Pre-training](https://mlsanity.com/paper/2305.07920/) (2023)
- [Hallucinations and Key Information Extraction in Medical Texts: A Comprehensive Assessment of Open-Source Large Language Models](https://mlsanity.com/paper/2504.19061/) (2025)

## Built on by (most cited first, 1 of 1)

- [Trustworthy Medical Imaging with Large Language Models: A Study of Hallucinations Across Modalities](https://mlsanity.com/paper/2508.07031/) (2025)

## Abstract (arXiv)

This study presents a multimodal AI framework designed for precisely classifying medical diagnostic images. Utilizing publicly available datasets, the proposed system compares the strengths of convolutional neural networks (CNNs) and different large language models (LLMs). This in-depth comparative analysis highlights key differences in diagnostic performance, execution efficiency, and environmental impacts. Model evaluation was based on accuracy, F1-score, average execution time, average energy consumption, and estimated CO2 emission. The findings indicate that although CNN-based models can outperform various multimodal techniques that incorporate both images and contextual information, applying additional filtering on top of LLMs can lead to substantial performance gains. These findings highlight the transformative potential of multimodal AI systems to enhance the reliability, efficiency, and scalability of medical diagnostics in clinical settings.