# Comparative Analysis of Large Language Models in Healthcare

arXiv: https://arxiv.org/abs/2604.10316 · HTML: https://mlsanity.com/paper/2604.10316/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 50 references that are in the map.
- First posted on arXiv 11 Apr 2026.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [Capabilities of GPT-4 on Medical Challenge Problems](https://mlsanity.com/paper/2303.13375/) (2023)
- [MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering](https://mlsanity.com/paper/2203.14371/) (2022)
- [ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledge](https://mlsanity.com/paper/2303.14070/) (2023)
- [A Framework for Human Evaluation of Large Language Models in Healthcare Derived from Literature Review](https://mlsanity.com/paper/2405.02559/) (2024)
- [Publicly Shareable Clinical Large Language Model Built on Synthetic Clinical Notes](https://mlsanity.com/paper/2309.00237/) (2023)
- [Radiology-Llama2: Best-in-Class Large Language Model for Radiology](https://mlsanity.com/paper/2309.06419/) (2023)
- ["I think this is the most disruptive technology": Exploring Sentiments of ChatGPT Early Adopters using Twitter Data](https://mlsanity.com/paper/2212.05856/) (2022)
- [On the Impact of Fine-Tuning on Chain-of-Thought Reasoning](https://mlsanity.com/paper/2411.15382/) (2024)
- [ChatNVD: Advancing Cybersecurity Vulnerability Assessment with Large Language Models](https://mlsanity.com/paper/2412.04756/) (2024)
- [RegimeFolio: A Regime Aware ML System for Sectoral Portfolio Optimization in Dynamic Markets](https://mlsanity.com/paper/2510.14986/) (2025)
- [A Review on C3I Systems' Security: Vulnerabilities, Attacks, and Countermeasures](https://mlsanity.com/paper/2104.11906/) (2021)
- [Smart HPA: A Resource-Efficient Horizontal Pod Auto-scaler for Microservice Architectures](https://mlsanity.com/paper/2403.07909/) (2024)
- [Robust Partial Least Squares Using Low Rank and Sparse Decomposition](https://mlsanity.com/paper/2407.06936/) (2024)
- [A Survey on Immersive Cyber Situational Awareness Systems](https://mlsanity.com/paper/2408.07456/) (2024)
- [Machine Learning Driven Smishing Detection Framework for Mobile Security](https://mlsanity.com/paper/2412.09641/) (2024)
- [The Future of AI: Exploring the Potential of Large Concept Models](https://mlsanity.com/paper/2501.05487/) (2025)
- [Co-Evolutionary Defence of Active Directory Attack Graphs via GNN-Approximated Dynamic Programming](https://mlsanity.com/paper/2505.11710/) (2025)
- [Resilient Auto-Scaling of Microservice Architectures with Efficient Resource Management](https://mlsanity.com/paper/2506.05693/) (2025)
- [Microservice Vulnerability Analysis: A Literature Review with Empirical Insights](https://mlsanity.com/paper/2408.03960/) (2024)
- [SCALAR: Self-Calibrating Adaptive Latent Attention Representation Learning](https://mlsanity.com/paper/2510.16474/) (2025)
- [Australian Bushfire Intelligence with AI-Driven Environmental Analytics](https://mlsanity.com/paper/2601.06105/) (2026)
- [Explainable Autonomous Cyber Defense using Adversarial Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2604.04442/) (2026)

## Built on by (most cited first, 1 of 1)

- [AgenticVM: Agentic AI for Adaptive Software Vulnerability Management](https://mlsanity.com/paper/2605.01739/) (2026)

## Abstract (arXiv)

Background: Large Language Models (LLMs) are transforming artificial intelligence applications in healthcare due to their ability to understand, generate, and summarize complex medical text. They offer valuable support to clinicians, researchers, and patients, yet their deployment in high-stakes clinical environments raises critical concerns regarding accuracy, reliability, and patient safety. Despite substantial attention in recent years, standardized benchmarking of LLMs for medical applications has been limited. Objective: This study addresses the need for a standardized comparative evaluation of LLMs in medical settings. Method: We evaluate multiple models, including ChatGPT, LLaMA, Grok, Gemini, and ChatDoctor, on core medical tasks such as patient note summarization and medical question answering, using the open-access datasets, MedMCQA, PubMedQA, and Asclepius, and assess performance through a combination of linguistic and task-specific metrics. Results: The results indicate that domain-specific models, such as ChatDoctor, excel in contextual reliability, producing medically accurate and semantically aligned text, whereas general-purpose models like Grok and LLaMA perform better in structured question-answering tasks, demonstrating higher quantitative accuracy. This highlights the complementary strengths of domain-specific and general-purpose LLMs depending on the medical task. Conclusion: Our findings suggest that LLMs can meaningfully support medical professionals and enhance clinical decision-making; however, their safe and effective deployment requires adherence to ethical standards, contextual accuracy, and human oversight in relevant cases. These results underscore the importance of task-specific evaluation and cautious integration of LLMs into healthcare workflows.