mlsanityData to 14 Oct 2026

Expert Evaluation of Clinical AI Tools on Real Point-of-Care Clinical Queries

arXiv 2606.28960 ↗First posted 27 Jun 2026cs.AI q-bio.QM stat.APcited by 5

Not on the map. Why?

Abstract (arXiv)

Physicians now pose millions of clinical questions to AI tools each week, yet these tools are evaluated largely on hypothetical or exam-style questions, not those actually asked in practice. We report a blinded evaluation built on 620 Real-world Point-Of-Care Queries (Real-POCQi) submitted to the OpenEvidence (OE) platform by physicians spanning 30 specialties, as well as 187 questions from HealthBench. 149 practicing physicians across 36 states made head-to-head comparisons between answers from three frontier general-purpose models (Claude Opus 4.8, Gemini 3.1 Pro, and GPT-5.5) and a specialized clinical tool (OE), with graders matched to each question's specialty. When comparing answers along five dimensions relevant to clinical decision support -- accuracy, clinical utility, source quality, verifiability, & completeness -- physicians scored the specialized tool highest on all axes; in the primary analysis on Real-POCQi, win differences (margins between win and loss rates) ranged from 25 to 39 percentage points (p<0.001). Results remained consistent in sensitivity analyses stratifying by citation display, answer length, OE-user status, and Real-POCQi versus HealthBench. In parallel, LLM judges were found to systematically differ from expert judges, though both generally agreed on the best model. These findings underscore two conclusions: (i) AI tool evaluations should reflect real-world query distributions and use expert judges that mirror the specialization defining modern medicine and (ii) the consistent advantage of the specialized tool over general-purpose models does not necessarily mean that the latter cannot serve similar purposes, but that targeted engineering and customization can yield meaningful gains in performance for its users. We release Real-POCQi as a public benchmark, as well as the prespecified statistical analysis for reproducing results of this study.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 18 of its 31 references that are in the map. First posted on arXiv 27 Jun 2026.

Builds on

18 of 31 references.

References · 18

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  3. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,356
  4. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  5. 2022BERTopic: Neural topic modeling with a class-based TF-IDF procedurecited by 801
  6. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 742
  7. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 429
  8. 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 326
  9. 2024WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wildcited by 218
  10. 2023Benchmarking Cognitive Biases in Large Language Models as Evaluatorscited by 150
  11. 2024Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluationscited by 148
  12. 2025Neither Valid nor Reliable? Investigating the Use of LLMs as Judgescited by 39
6 more
  1. 2025Medical Large Language Model Benchmarks Should Prioritize Construct Validitycited by 38
  2. 2024Prediction-Powered Ranking of Large Language Modelscited by 29
  3. 2026Human-AI Co-design for Clinical Prediction Modelscited by 9
  4. 2026JudgmentBench: Comparing Rubric and Preference Evaluation for Quality Assessmentcited by 5
  5. 2025LLMs Judging LLMs: A Simplex Perspectivecited by 4
  6. 2026Statistical inference with win statistics in cluster-randomized trials with hierarchical composite outcomescited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026Agentic systems for breast cancer treatment recommendationscited by 0
  2. 2026Judge-dependent safety gains and model-specific helpfulness costs of evidence-sufficiency prompting in clinical LLMscited by 0
  3. 2026Scaling Clinical Judgment to Evaluate Medical AIcited by 0
  4. 2026Verifiable by Construction: Claim-Level Evaluation of Verbatim Citation in Clinical Question Answeringcited by 0
  5. 2026Clinician use of language models diverges from how the models are evaluatedcited by 0