mlsanityData to 14 Oct 2026

Automatic Evaluation of Healthcare LLMs Beyond Question-Answering

arXiv 2502.06666 ↗First posted 10 Feb 2025cs.CL cs.AIcited by 20

Not on the map. Why?

Abstract (arXiv)

Current Large Language Models (LLMs) benchmarks are often based on open-ended or close-ended QA evaluations, avoiding the requirement of human labor. Close-ended measurements evaluate the factuality of responses but lack expressiveness. Open-ended capture the model's capacity to produce discourse responses but are harder to assess for correctness. These two approaches are commonly used, either independently or together, though their relationship remains poorly understood. This work is focused on the healthcare domain, where both factuality and discourse matter greatly. It introduces a comprehensive, multi-axis suite for healthcare LLM evaluation, exploring correlations between open and close benchmarks and metrics. Findings include blind spots and overlaps in current methodologies. As an updated sanity check, we release a new medical benchmark --CareQA-- with both open and closed variants. Finally, we propose a novel metric for open-ended evaluations -- Relaxed Perplexity -- to mitigate the identified limitations.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Builds on 25 of its 42 references that are in the map. First posted on arXiv 10 Feb 2025.

Builds on

25 of 42 references.

References · 25

  1. 2024Qwen2 Technical Reportcited by 2,120
  2. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  3. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,356
  4. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  5. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  6. 2024Yi: Open Foundation Models by 01.AIcited by 719
  7. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 486
  8. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 446
  9. 2016Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Modelscited by 301
  10. 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 263
  11. 2022Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matterscited by 222
  12. 2024Chain-of-Thought Reasoning Without Promptingcited by 210
13 more
  1. 2024A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicitycited by 185
  2. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 163
  3. 2024When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboardscited by 137
  4. 2023A Survey of Large Language Models in Medicine: Progress, Application, and Challengecited by 124
  5. 2023ACI-BENCH: a Novel Ambient Clinical Intelligence Dataset for Benchmarking Automatic Visit Note Generationcited by 118
  6. 2024Med42-v2: A Suite of Clinical LLMscited by 90
  7. 2022BioRED: A Rich Biomedical Relation Extraction Datasetcited by 85
  8. 2024Can multiple-choice questions really be useful in detecting the abilities of LLMs?cited by 69
  9. 2022Evaluating Unsupervised Text Classification: Zero-shot and Similarity-based Approachescited by 39
  10. 2023Creating Trustworthy LLMs: Dealing with Hallucinations in Healthcare AIcited by 33
  11. 2024MEDIC: Comprehensive Evaluation of Leading Indicators for LLM Safety and Utility in Clinical Applicationscited by 25
  12. 2024OLAPH: Improving Factuality in Biomedical Long-form Question Answeringcited by 20
  13. 2024MedConceptsQA: Open Source Medical Concepts QA Benchmarkcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 10; 2026: 10.

20 papers.

Sort:

By area:

  1. 2025FlowerTune: A Cross-Domain Benchmark for Federated Fine-Tuning of Large Language Modelscited by 11
  2. 2025Fleming-R1: Toward Expert-Level Medical Reasoning via Reinforcement Learningcited by 10
  3. 2025Gaperon: A Peppered English-French Generative Language Model Suitecited by 9
  4. 2025Essential-Web v1.0: 24T tokens of organized web datacited by 8
  5. 2025SynapseRoute: An Auto-Route Switching Framework on Dual-State Large Language Modelcited by 7
  6. 2025Understanding the Effects of Domain Finetuning on LLMscited by 4
  7. 2026RubricsTree: Scalable and Evolving Open-Ended Evaluation of Personal Health Agents across Health Memory and Medical Skillscited by 4
  8. 2025Beyond the Rubric: Cultural Misalignment in LLM Benchmarks for Sexual and Reproductive Healthcited by 2
  9. 2026Assessing Large Language Models for Medical QA: Zero-Shot and LLM-as-a-Judge Evaluationcited by 2
  10. 2026Medmarks: A Comprehensive Open-Source LLM Benchmark Suite for Medical Taskscited by 2
  11. 2026Demystifying Numerical Instability in LLM Inference: Achieving Reproducible Inference for Mission-Critical Tasks with HEALcited by 2
  12. 2025Toward Revealing Nuanced Biases in Medical LLMscited by 1
8 more
  1. 2025Filling in the Clinical Gaps in Benchmark: Case for HealthBench for the Japanese medical systemcited by 1
  2. 2026Aloe-Vision: Robust Vision-Language Models for Healthcarecited by 1
  3. 2026MTDiag: A Multi-Turn Diagnostic Dataset Towards Clinically Meaningful LLM Evaluationcited by 1
  4. 2026CPR for LLMs: Critical-Point Routing against Catastrophic Forgetting in Domain Adaptationcited by 1
  5. 2025Investigating LLM Capabilities on Long Context Comprehension for Medical Question Answeringcited by 0
  6. 2026Knowledge-Free Correlated Agreement for Incentivizing Federated Learningcited by 0
  7. 2026PetQA: Benchmarking Veterinary Knowledge and Clinical Reasoningcited by 0
  8. 2026Constructing and Evaluating Clinical Reasoning Trajectories for Medical Agentcited by 0