mlsanityData to 7 Oct 2026

Large Language Models in the Clinic: A Comprehensive Benchmark

arXiv 2405.00716 ↗First posted 25 Apr 2024cs.CL cs.AIcited by 12

Not on the map. Why?

Abstract (arXiv)

The adoption of large language models (LLMs) to assist clinicians has attracted remarkable attention. Existing works mainly adopt the close-ended question-answering (QA) task with answer options for evaluation. However, many clinical decisions involve answering open-ended questions without pre-set options. To better understand LLMs in the clinic, we construct a benchmark ClinicBench. We first collect eleven existing datasets covering diverse clinical language generation, understanding, and reasoning tasks. Furthermore, we construct six novel datasets and clinical tasks that are complex but common in real-world practice, e.g., open-ended decision-making, long document processing, and emerging drug analysis. We conduct an extensive evaluation of twenty-two LLMs under both zero-shot and few-shot settings. Finally, we invite medical experts to evaluate the clinical usefulness of LLMs. The benchmark data is available at https://github.com/AI-in-Health/ClinicBench.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 29 of its 70 references that are in the map. First posted on arXiv 25 Apr 2024.

Builds on

29 of 70 references.

References · 29

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  8. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  9. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  10. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 996
  11. 2023Towards Expert-Level Medical Question Answering with Large Language Modelscited by 382
  12. 2023ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledgecited by 324
17 more
  1. 2023Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicinecited by 319
  2. 2024BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domainscited by 299
  3. 2019Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasetscited by 298
  4. 2023Towards Generalist Biomedical AIcited by 286
  5. 2023MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Datacited by 282
  6. 2023Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarizationcited by 244
  7. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 162
  8. 2021Exploring and Distilling Posterior and Prior Knowledge for Radiology Report Generationcited by 160
  9. 2023Does Synthetic Data Generation of LLMs Help Clinical Text Mining?cited by 151
  10. 2023A Survey of Large Language Models in Medicine: Progress, Application, and Challengecited by 123
  11. 2023Almanac: Retrieval-Augmented Language Models for Clinical Medicinecited by 122
  12. 2023Zhongjing: Enhancing the Chinese Medical Capabilities of Large Language Model through Expert Feedback and Real-world Multi-turn Dialoguecited by 118
  13. 2023AlpaCare:Instruction-tuned Large Language Models for Medical Applicationcited by 88
  14. 2023ClinicalGPT: Large Language Models Finetuned with Diverse Medical Data and Comprehensive Evaluationcited by 85
  15. 2021Contrastive Attention for Automatic Chest X-ray Report Generationcited by 71
  16. 2023An Iterative Optimizing Framework for Radiology Report Summarization with ChatGPTcited by 57
  17. 2023A Comprehensive Evaluation of Large Language Models on Benchmark Biomedical Text Processing Taskscited by 36

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 7; 2026: 4.

12 papers.

  1. 2025MedHELM: Holistic Evaluation of Large Language Models for Medical Taskscited by 60
  2. 2024From Text to Multimodality: Exploring the Evolution and Impact of Large Language Models in Medical Practicecited by 8
  3. 2025CUPCase: Clinically Uncommon Patient Cases and Diagnoses Datasetcited by 6
  4. 2025Susceptibility of Large Language Models to User-Driven Factors in Medical Queriescited by 6
  5. 2026Possible or Definite? A Benchmark for Evaluating Diagnostic Uncertainty Preservation in Clinical Textcited by 4
  6. 2025A Principle-based Framework for the Development and Evaluation of Large Language Models for Health and Wellnesscited by 3
  7. 2026EHRBench: An Automated and Reliable EHR-based Benchmark for Clinical Decision Making with LLMscited by 2
  8. 2025Multi-Task Learning for Extracting Menstrual Characteristics from Clinical Notescited by 0
  9. 2025ImmunoFOMO: Are Language Models missing what oncologists see?cited by 0
  10. 2025Mirror, Mirror on the Wall -- Which is the Best Model of Them All?cited by 0
  11. 2026Towards the Next Frontier of LLMs, Training on Private Data: A Cross-Domain Benchmark for Federated Fine-Tuningcited by 0
  12. 2026A Task-Oriented Multi-Agent Framework for Complex Wearable Health Analysiscited by 0