mlsanityData to 14 Oct 2026

The widening evaluation gap in medical large language model research 2023 to 2026

arXiv 2609.11770 ↗First posted 10 Sep 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Large language models are superseded every few quarters; clinical evidence takes years. We asked whether medical research is keeping pace with the systems it evaluates. PubMed returned 11,628 records for January 2023 to June 2026 across fourteen clinical domains, growing 45-fold; 2.5% used a randomised, controlled or prospective design. Evaluation lag, from a study's newest named model release to its own publication, widened from 1.33 to 6.08 quarters. Because discontinued models age mechanically, we benchmarked this against a counterfactual holding model composition fixed: migration to newer systems offset only 56% of the drift (95% CI 50-65). Randomised trials evaluated models a median 4.6 quarters older than other designs (P = 3 x 10^-19), yet among studies naming a model still under development no design differed from any other; 62% of randomised trials evaluated a discontinued family. Rigour and currency are in tension, and that tension reflects model selection rather than research timelines.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 4 of its 38 references that are in the map. First posted on arXiv 10 Sep 2026.

Builds on

4 of 38 references.

References · 4

  1. 2025MedGemma Technical Reportcited by 436
  2. 2023Towards Accurate Differential Diagnosis with Large Language Modelscited by 116
  3. 2024When combinations of humans and AI are useful: A systematic review and meta-analysiscited by 106
  4. 2024A Framework for Human Evaluation of Large Language Models in Healthcare Derived from Literature Reviewcited by 66

Built on by

Not cited yet.