mlsanityData to 14 Oct 2026

HealthBench: Evaluating Large Language Models Towards Improved Human Health

arXiv 2505.08775 ↗First posted 13 May 2025cs.CLcited by 429

Not on the map. Why?

Cited by 81 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present HealthBench, an open-source benchmark measuring the performance and safety of large language models in healthcare. HealthBench consists of 5,000 multi-turn conversations between a model and an individual user or healthcare professional. Responses are evaluated using conversation-specific rubrics created by 262 physicians. Unlike previous multiple-choice or short-answer benchmarks, HealthBench enables realistic, open-ended evaluation through 48,562 unique rubric criteria spanning several health contexts (e.g., emergencies, transforming clinical data, global health) and behavioral dimensions (e.g., accuracy, instruction following, communication). HealthBench performance over the last two years reflects steady initial progress (compare GPT-3.5 Turbo's 16% to GPT-4o's 32%) and more rapid recent improvements (o3 scores 60%). Smaller models have especially improved: GPT-4.1 nano outperforms GPT-4o and is 25 times cheaper. We additionally release two HealthBench variations: HealthBench Consensus, which includes 34 particularly important dimensions of model behavior validated via physician consensus, and HealthBench Hard, where the current top score is 32%. We hope that HealthBench grounds progress towards model development and applications that benefit human health.

Key facts. Cited by 429 papers in the map (data through 14 Oct 2026). Builds on 27 of its 52 references that are in the map. First posted on arXiv 13 May 2025.

Builds on

27 of 52 references.

References · 27

  1. 2024GPT-4o System Cardcited by 4,588
  2. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  3. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  4. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,356
  5. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
  6. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  7. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 638
  8. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 629
  9. 2022BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Miningcited by 538
  10. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 461
  11. 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 392
  12. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 368
15 more
  1. 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 323
  2. 2025PaperBench: Evaluating AI's Ability to Replicate AI Researchcited by 321
  3. 2021RadGraph: Extracting Clinical Entities and Relations from Radiology Reportscited by 319
  4. 2023Towards Generalist Biomedical AIcited by 289
  5. 2024Capabilities of Gemini Models in Medicinecited by 237
  6. 2024WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wildcited by 218
  7. 2025MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMscited by 180
  8. 2023Almanac: Retrieval-Augmented Language Models for Clinical Medicinecited by 123
  9. 2023Large Language Models are Few-Shot Health Learnerscited by 118
  10. 2023Towards Accurate Differential Diagnosis with Large Language Modelscited by 116
  11. 2023MedAlign: A Clinician-Generated Dataset for Instruction Following with Electronic Medical Recordscited by 56
  12. 2024A Toolbox for Surfacing Health Equity Harms and Biases in Large Language Modelscited by 45
  13. 2023Evaluation of GPT-3.5 and GPT-4 for supporting real-world information needs in healthcare deliverycited by 30
  14. 2024Towards a Personal Health Large Language Modelcited by 27
  15. 2015Scaling-up Empirical Risk Minimization: Optimization of Incomplete U-statisticscited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 107; 2026: 322.

429 papers.

Sort:

By area:

  1. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,386
  2. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 366
  3. 2025Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoningcited by 230
  4. 2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignmentcited by 132
  5. 2025Reinforcement Learning with Rubric Anchorscited by 113
  6. 2025MedHELM: Holistic Evaluation of Large Language Models for Medical Taskscited by 64
  7. 2026PostTrainBench: Can LLM Agents Automate LLM Post-Training?cited by 64
  8. 2025Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for General LLM Reasoningcited by 63
  9. 2026Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Taskscited by 62
  10. 2025Sequential Diagnosis with Language Modelscited by 60
  11. 2025Baichuan-M2: Scaling Medical Capability with Large Verifier Systemcited by 59
  12. 2025Every Step Evolves: Scaling Reinforcement Learning for Trillion-Scale Thinking Modelcited by 55
38 more
  1. 2026Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Trainingcited by 54
  2. 2025Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Trainingcited by 53
  3. 2025VitaBench: Benchmarking LLM Agents with Versatile Interactive Tasks in Real-world Applicationscited by 53
  4. 2025Online Rubrics Elicitation from Pairwise Comparisonscited by 46
  5. 2025Answer Matching Outperforms Multiple Choice for Language Model Evaluationcited by 41
  6. 2025A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontierscited by 37
  7. 2025Beyond GPT-5: Making LLMs Cheaper and Better via Performance-Efficiency Optimized Routingcited by 34
  8. 2026DeepResearch Bench II: Diagnosing Deep Research Agents via Rubrics from Expert Reportscited by 34
  9. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 33
  10. 2025PRBench: Large-Scale Expert Rubrics for Evaluating High-Stakes Professional Reasoningcited by 32
  11. 2026STEP3-VL-10B Technical Reportcited by 32
  12. 2025Auto-Rubric: Learning From Implicit Weights to Explicit Rubrics for Reward Modelingcited by 30
  13. 2025Beyond Seeing: Evaluating Multimodal LLMs on Tool-Enabled Image Perception, Transformation, and Reasoningcited by 29
  14. 2025A Novel Evaluation Benchmark for Medical LLMs: Illuminating Safety and Effectiveness in Clinical Domainscited by 28
  15. 2026RubricBench: Aligning Model-Generated Rubrics with Human Standardscited by 28
  16. 2025Real-World Doctor Agent with Proactive Consultation through Multi-Agent Reinforcement Learningcited by 27
  17. 2025AlgoTune: Can Language Models Speed Up General-Purpose Numerical Programs?cited by 27
  18. 2026Reinforcing Chain-of-Thought Reasoning with Self-Evolving Rubricscited by 26
  19. 2026HealthBench Professional: Evaluating Large Language Models on Real Clinician Chatscited by 26
  20. 2026Agentic Rubrics as Contextual Verifiers for SWE Agentscited by 25
  21. 2025MedAgentGym: A Scalable Agentic Training Environment for Code-Centric Reasoning in Biomedical Data Sciencecited by 23
  22. 2025MoReBench: Evaluating Procedural and Pluralistic Moral Reasoning in Language Models, More than Outcomescited by 23
  23. 2025DAComp: Benchmarking Data Agents across the Full Data Intelligence Lifecyclecited by 23
  24. 2025ER-Reason: A Benchmark Dataset for LLM Clinical Reasoning in the Emergency Roomcited by 22
  25. 2025Reward and Guidance through Rubrics: Promoting Exploration to Improve Multi-Domain Reasoningcited by 21
  26. 2026PhysicianBench: Evaluating LLM Agents in Real-World EHR Environmentscited by 21
  27. 2025Audio MultiChallenge: A Multi-Turn Evaluation of Spoken Dialogue Systems on Natural Human Interactioncited by 19
  28. 2025Direct Reasoning Optimization: Token-Level Reasoning Reflectivity Meets Rubric Gates for Unverifiable Taskscited by 18
  29. 2025Large language models provide unsafe answers to patient-posed medical questionscited by 18
  30. 2025Compute as Teacher: Turning Inference Compute Into Reference-Free Supervisioncited by 18
  31. 2025Mitigating LLM Hallucination via Behaviorally Calibrated Reinforcement Learningcited by 18
  32. 2026Agentic Aggregation for Parallel Scaling of Long-Horizon Agentic Taskscited by 18
  33. 2026GUI Agents for Continual Game Generationcited by 18
  34. 2025LLMs Don't Know Their Own Decision Boundaries: The Unreliability of Self-Generated Counterfactual Explanationscited by 16
  35. 2026Reproducing, Analyzing, and Detecting Reward Hacking in Rubric-Based Reinforcement Learningcited by 16
  36. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 15
  37. 2026MedMO: Grounding and Understanding Multimodal Large Language Model for Medical Imagescited by 15
  38. 2025Doctor-R1: Mastering Clinical Inquiry with Experiential Agentic Reinforcement Learningcited by 14

50 most cited of 429 All 429 (JSON)