# Generalist Large Language Models Outperform Clinical Tools on Medical Benchmarks

arXiv: https://arxiv.org/abs/2512.01191 · HTML: https://mlsanity.com/paper/2512.01191/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 10 references that are in the map.
- First posted on arXiv 1 Dec 2025.

## Builds on (most cited first, 7 of 7)

- [What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams](https://mlsanity.com/paper/2009.13081/) (2020)
- [HealthBench: Evaluating Large Language Models Towards Improved Human Health](https://mlsanity.com/paper/2505.08775/) (2025)
- [ClashEval: Quantifying the tug-of-war between an LLM's internal prior and external evidence](https://mlsanity.com/paper/2404.10198/) (2024)
- [Medical large language models are easily distracted](https://mlsanity.com/paper/2504.01201/) (2025)
- [MedMobile: A mobile-sized language model with clinical capabilities](https://mlsanity.com/paper/2410.09019/) (2024)
- [CNS-Obsidian: A Neurosurgical Vision-Language Model Built From Scientific Publications](https://mlsanity.com/paper/2502.19546/) (2025)
- [Generalist Foundation Models Are Not Clinical Enough for Hospital Operations](https://mlsanity.com/paper/2511.13703/) (2025)

## Built on by (most cited first, 2 of 2)

- [Large Language Models Predict Functional Outcomes after Acute Ischemic Stroke](https://mlsanity.com/paper/2602.10119/) (2026)
- [OpenBioRQ: Unsolved Biomedical Research Questions for Agents](https://mlsanity.com/paper/2606.21959/) (2026)

## Abstract (arXiv)

Specialized clinical AI assistants are rapidly entering medical practice, often framed as safer or more reliable than general-purpose large language models (LLMs). Yet, unlike frontier models, these clinical tools are rarely subjected to independent, quantitative evaluation, creating a critical evidence gap despite their growing influence on diagnosis, triage, and guideline interpretation. We assessed two widely deployed clinical AI systems (OpenEvidence and UpToDate Expert AI) against three state-of-the-art generalist LLMs (GPT-5, Gemini 3 Pro, and Claude Sonnet 4.5) using a 1,000-item mini-benchmark combining MedQA (medical knowledge) and HealthBench (clinician-alignment) tasks. Generalist models consistently outperformed clinical tools, with GPT-5 achieving the highest scores, while OpenEvidence and UpToDate demonstrated deficits in completeness, communication quality, context awareness, and systems-based safety reasoning. These findings reveal that tools marketed for clinical decision support may often lag behind frontier LLMs, underscoring the urgent need for transparent, independent evaluation before deployment in patient-facing workflows.