mlsanityData to 14 Oct 2026

ClinicalBench: Can LLMs Beat Traditional ML Models in Clinical Prediction?

arXiv 2411.06469 ↗First posted 10 Nov 2024cs.CLcited by 32

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) hold great promise to revolutionize current clinical systems for their superior capacities on medical text processing tasks and medical licensing exams. Meanwhile, traditional ML models such as SVM and XGBoost have still been mainly adopted in clinical prediction tasks. An emerging question is: Can LLMs beat traditional ML models in clinical prediction? Thus, we build a new benchmark ClinicalBench to comprehensively study the clinical predictive modeling capacities of both general-purpose and medical LLMs, and compare them with traditional ML models. ClinicalBench embraces three common clinical prediction tasks, two databases, 14 general-purpose LLMs, 8 medical LLMs, and 11 traditional ML models. Through extensive empirical investigation, we discover that both general-purpose and medical LLMs, even with different model scales, diverse prompting or fine-tuning strategies, still cannot beat traditional ML models in clinical prediction yet, shedding light on their potential deficiency in clinical reasoning and decision-making. We call for caution when practitioners adopt LLMs in clinical applications. ClinicalBench can be utilized to bridge the gap between LLMs' development for healthcare and real-world clinical practice.

Key facts. Cited by 32 papers in the map (data through 14 Oct 2026). Builds on 31 of its 72 references that are in the map. First posted on arXiv 10 Nov 2024.

Builds on

31 of 72 references.

References · 31

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  4. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  5. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,356
  6. 2022A Survey on In-context Learningcited by 721
  7. 2024Yi: Open Foundation Models by 01.AIcited by 719
  8. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 716
  9. 2023Instruction Tuning for Large Language Models: A Surveycited by 623
  10. 2024A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applicationscited by 431
  11. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 238
  12. 2024Large Language Models for Data Annotation and Synthesis: A Surveycited by 205
19 more
  1. 2019MIMIC-Extract: A Data Extraction, Preprocessing, and Representation Pipeline for MIMIC-IIIcited by 122
  2. 2023CMB: A Comprehensive Medical Benchmark in Chinesecited by 115
  3. 2024Benchmarking Large Language Models on Answering and Explaining Challenging Medical Questionscited by 109
  4. 2023Benchmarking Large Language Models on CMExam -- A Comprehensive Chinese Medical Exam Datasetcited by 91
  5. 2024Med42-v2: A Suite of Clinical LLMscited by 90
  6. 2021Pre-trained Language Models in Biomedical Domain: A Systematic Surveycited by 87
  7. 2023Evaluating GPT-4 and ChatGPT on Japanese Medical Licensing Examinationscited by 63
  8. 2024Comprehensive Exploration of Synthetic Data Generation: A Surveycited by 60
  9. 2023GraphCare: Enhancing Healthcare Predictions with Personalized Knowledge Graphscited by 58
  10. 2024RareBench: Can LLMs Serve as Rare Diseases Specialists?cited by 45
  11. 2024MedExpQA: Multilingual Benchmarking of Large Language Models for Medical Question Answeringcited by 43
  12. 2023A Comprehensive Evaluation of Large Language Models on Benchmark Biomedical Text Processing Taskscited by 36
  13. 2023MedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Modelscited by 31
  14. 2024Hidden flaws behind expert-level accuracy of multimodal GPT-4 vision in medicinecited by 31
  15. 2024KorMedMCQA: Multi-Choice Question Answering Benchmark for Korean Healthcare Professional Licensing Examinationscited by 22
  16. 2024To Cool or not to Cool? Temperature Network Meets Large Foundation Models via DROcited by 11
  17. 2024MedREQAL: Examining Medical Knowledge Recall of Large Language Models via Question Answeringcited by 9
  18. 2024MedConceptsQA: Open Source Medical Concepts QA Benchmarkcited by 6
  19. 2024Evaluating Large Language Models for Public Health Classification and Extraction Taskscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 20; 2026: 12.

32 papers.

Sort:

By area:

  1. 2025TxAgent: An AI Agent for Therapeutic Reasoning Across a Universe of Toolscited by 59
  2. 2025A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontierscited by 37
  3. 2025Large Language Models are Powerful Electronic Health Record Encoderscited by 26
  4. 2025A Modular Approach for Clinical SLMs Driven by Synthetic Data with Pre-Instruction Tuning, Model Merging, and Clinical-Tasks Alignmentcited by 20
  5. 2026Beyond IID: How General Are Tabular Foundation Models, Really?cited by 20
  6. 2025A foundation model for human-AI collaboration in medical literature miningcited by 11
  7. 2025A Collection of Innovations in Medical AI for patient records in 2024cited by 7
  8. 2025MoMA: A Mixture-of-Multimodal-Agents Architecture for Enhancing Clinical Prediction Modellingcited by 7
  9. 2026EHR-RAG: Bridging Long-Horizon Structured Electronic Health Records and Large Language Models via Enhanced Retrieval-Augmented Generationcited by 6
  10. 2025Enhancing Test-Time Scaling of Large Language Models with Hierarchical Retrieval-Augmented MCTScited by 5
  11. 2026MedEinst: Benchmarking the Einstellung Effect in Medical LLMs through Counterfactual Differential Diagnosiscited by 4
  12. 2025Toward Better EHR Reasoning in LLMs: Reinforcement Learning with Expert Attention Guidancecited by 3
20 more
  1. 2025Count-Based Approaches Remain Strong: A Benchmark Against Transformer and LLM Pipelines on Structured EHRcited by 3
  2. 2025Interpreting LLMs as Credit Risk Classifiers: Do Their Feature Explanations Align with Classical ML?cited by 2
  3. 2025TCM-Eval: An Expert-Level Dynamic and Extensible Benchmark for Traditional Chinese Medicinecited by 2
  4. 2026GraphWalker: Patient Analogy Meets Information Gain for Clinical Reasoning with Large Language Modelscited by 2
  5. 2026From Token to Token Pair: Efficient Prompt Compression for Large Language Models in Clinical Predictioncited by 2
  6. 2026The Path to Self-Evolving Clinical Systems: Scaling Medical Agents from Assistance to Autonomycited by 2
  7. 2025BehaviorSFT: Behavioral Token Conditioning for Clinical Agents Across the Proactivity Spectrumcited by 1
  8. 2025Beyond MedQA: Towards Real-world Clinical Decision Making in the Era of LLMscited by 1
  9. 2025Grounded by Experience: Generative Healthcare Prediction Augmented with Hierarchical Agentic Retrievalcited by 1
  10. 2025Review GIDE -- Restaurant Review Gastrointestinal Illness Detection and Extraction with Large Language Modelscited by 0
  11. 2025Holistic Artificial Intelligence in Medicine; improved performance and explainabilitycited by 0
  12. 2025A Locally Executable AI System for Improving Preoperative Patient Communication: A Multi-Domain Clinical Evaluationcited by 0
  13. 2025NurseLLM: The First Specialized Language Model for Nursingcited by 0
  14. 2025Counting Clues: A Lightweight Probabilistic Baseline Can Match an LLMcited by 0
  15. 2026Learning to Select Source-Traceable Evidence for Language-Model Prediction from Irregular Clinical Time Seriescited by 0
  16. 2026Are LLMs Ready to Assist Physicians? PhysAssistBench for Interactive Doctor-Patient-EHR Assistancecited by 0
  17. 2026MIRA-Ev:A Benchmark for Granular Evidence Detection and Relational Reasoning in Clinical Examscited by 0
  18. 2026PatientAgentBench: A Benchmark Framework for Evaluating Patient-Facing Health AI Agentscited by 0
  19. 2026GLoC-EHR: Evidence-Cited Clinical Reasoning over Global Context and Local EHR Eventscited by 0
  20. 2026ActiveMedAgent: Cost-Aware Trajectory Learning for Multimodal Medical Diagnosiscited by 0