# CliBench: A Multifaceted and Multigranular Evaluation of Large Language Models for Clinical Decision Making

arXiv: https://arxiv.org/abs/2406.09923 · HTML: https://mlsanity.com/paper/2406.09923/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 82 references that are in the map.
- First posted on arXiv 14 Jun 2024.

## Builds on (most cited first, 21 of 21)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams](https://mlsanity.com/paper/2009.13081/) (2020)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [Capabilities of GPT-4 on Medical Challenge Problems](https://mlsanity.com/paper/2303.13375/) (2023)
- [MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering](https://mlsanity.com/paper/2203.14371/) (2022)
- [Towards Expert-Level Medical Question Answering with Large Language Models](https://mlsanity.com/paper/2305.09617/) (2023)
- [Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine](https://mlsanity.com/paper/2311.16452/) (2023)
- [Towards Accurate Differential Diagnosis with Large Language Models](https://mlsanity.com/paper/2312.00164/) (2023)
- [Clinical Camel: An Open Expert-Level Medical Language Model with Dialogue-Based Knowledge Encoding](https://mlsanity.com/paper/2305.12031/) (2023)
- [Publicly Shareable Clinical Large Language Model Built on Synthetic Clinical Notes](https://mlsanity.com/paper/2309.00237/) (2023)
- [Large Language Models are Clinical Reasoners: Reasoning-Aware Diagnosis Framework with Prompt-Generated Rationales](https://mlsanity.com/paper/2312.07399/) (2023)
- [DICE: Data-Efficient Clinical Event Extraction with Generative Models](https://mlsanity.com/paper/2208.07989/) (2022)
- [PMC-Patients: A Large-scale Dataset of Patient Summaries and Relations for Benchmarking Retrieval-based Clinical Decision Support Systems](https://mlsanity.com/paper/2202.13876/) (2022)
- [Can NLI Provide Proper Indirect Supervision for Low-resource Biomedical Relation Extraction?](https://mlsanity.com/paper/2212.10784/) (2022)
- [Mitigating Bias for Question Answering Models by Tracking Bias Influence](https://mlsanity.com/paper/2310.08795/) (2023)
- [STAR: Boosting Low-Resource Information Extraction by Structure-to-Text Data Generation with Large Language Models](https://mlsanity.com/paper/2305.15090/) (2023)
- [ChatGPT Assisting Diagnosis of Neuro-ophthalmology Diseases Based on Case Reports](https://mlsanity.com/paper/2309.12361/) (2023)

## Built on by (most cited first, 8 of 8)

- [MatSciBench: Benchmarking the Reasoning Ability of Large Language Models in Materials Science](https://mlsanity.com/paper/2510.12171/) (2025)
- [Memorize and Rank: Elevating Large Language Models for Clinical Diagnosis Prediction](https://mlsanity.com/paper/2501.17326/) (2025)
- [CLIMB: A Benchmark of Clinical Bias in Large Language Models](https://mlsanity.com/paper/2407.05250/) (2024)
- [A Collection of Innovations in Medical AI for patient records in 2024](https://mlsanity.com/paper/2503.05768/) (2025)
- [MedCTA: A Benchmark for Clinical Tool Agents](https://mlsanity.com/paper/2606.11702/) (2026)
- [RealICU: Do LLM Agents Understand Long-Context ICU Data? A Benchmark Beyond Behavior Imitation](https://mlsanity.com/paper/2605.13542/) (2026)
- [RxEval: A Prescription-Level Benchmark for Evaluating LLM Medication Recommendation](https://mlsanity.com/paper/2605.14543/) (2026)
- [ClinTraceBench: Source-Verifiable Longitudinal Clinical Reasoning over EHR-Derived Dialogues](https://mlsanity.com/paper/2609.01111/) (2026)

## Abstract (arXiv)

The integration of Artificial Intelligence (AI), especially Large Language Models (LLMs), into the clinical diagnosis process offers significant potential to improve the efficiency and accessibility of medical care. While LLMs have shown some promise in the medical domain, their application in clinical diagnosis remains underexplored, especially in real-world clinical practice, where highly sophisticated, patient-specific decisions need to be made. Current evaluations of LLMs in this field are often narrow in scope, focusing on specific diseases or specialties and employing simplified diagnostic tasks. To bridge this gap, we introduce CliBench, a novel benchmark developed from the MIMIC IV dataset, offering a comprehensive and realistic assessment of LLMs' capabilities in clinical diagnosis. This benchmark not only covers diagnoses from a diverse range of medical cases across various specialties but also incorporates tasks of clinical significance: treatment procedure identification, lab test ordering and medication prescriptions. Supported by structured output ontologies, CliBench enables a precise and multi-granular evaluation, offering an in-depth understanding of LLM's capability on diverse clinical tasks of desired granularity. We conduct a zero-shot evaluation of leading LLMs to assess their proficiency in clinical decision-making. Our preliminary results shed light on the potential and limitations of current LLMs in clinical settings, providing valuable insights for future advancements in LLM-powered healthcare.