mlsanityData to 7 Oct 2026

Evaluating Medical LLMs by Levels of Autonomy: A Survey Moving from Benchmarks to Applications

arXiv 2510.17764 ↗First posted 20 Oct 2025cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Medical Large language models achieve strong scores on standard benchmarks; however, the transfer of those results to safe and reliable performance in clinical workflows remains a challenge. This survey reframes evaluation through a levels-of-autonomy lens (L0-L3), spanning informational tools, information transformation and aggregation, decision support, and supervised agents. We align existing benchmarks and metrics with the actions permitted at each level and their associated risks, making the evaluation targets explicit. This motivates a level-conditioned blueprint for selecting metrics, assembling evidence, and reporting claims, alongside directions that link evaluation to oversight. By centering autonomy, the survey moves the field beyond score-based claims toward credible, risk-aware evidence for real clinical use.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 78 of its 160 references that are in the map. First posted on arXiv 20 Oct 2025.

Builds on

78 of 160 references.

References · 78

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  5. 2017On Calibration of Modern Neural Networkscited by 4,860
  6. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  7. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  8. 2024A Survey on LLM-as-a-Judgecited by 1,384
  9. 2018Datasheets for Datasetscited by 1,371
  10. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  11. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  12. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
38 more
  1. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
  2. 2018Model Cards for Model Reportingcited by 1,068
  3. 2017ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseasescited by 969
  4. 2017Selective Classification for Deep Neural Networkscited by 799
  5. 2018A Discourse-Aware Attention Model for Abstractive Summarization of Long Documentscited by 477
  6. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  7. 2019Measuring Calibration in Deep Learningcited by 443
  8. 2020SPECTER: Document-level Representation Learning using Citation-informed Transformerscited by 426
  9. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 354
  10. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 351
  11. 2019SelectiveNet: A Deep Neural Network with an Integrated Reject Optioncited by 333
  12. 2024Benchmarking Retrieval-Augmented Generation for Medicinecited by 326
  13. 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 314
  14. 2019Verified Uncertainty Calibrationcited by 307
  15. 2018GuacaMol: Benchmarking Models for De Novo Molecular Designcited by 294
  16. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 294
  17. 2018Molecular Sets (MOSES): A Benchmarking Platform for Molecular Generation Modelscited by 270
  18. 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 249
  19. 2024Self-Preference Bias in LLM-as-a-Judgecited by 248
  20. 2024Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMscited by 246
  21. 2024AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environmentscited by 215
  22. 2023MedCPT: Contrastive Pre-trained Transformers with Large-scale PubMed Search Logs for Zero-shot Biomedical Information Retrievalcited by 182
  23. 2023A Study of Generative Large Language Model for Medical Research and Healthcarecited by 160
  24. 2020MedDialog: Two Large-scale Medical Dialogue Datasetscited by 157
  25. 2023Large language models in healthcare and medical domain: A reviewcited by 126
  26. 2023ACI-BENCH: a Novel Ambient Clinical Intelligence Dataset for Benchmarking Automatic Visit Note Generationcited by 116
  27. 2024What Large Language Models Know and What People Think They Knowcited by 109
  28. 2024Are self-explanations from Large Language Models faithful?cited by 101
  29. 2021CBLUE: A Chinese Biomedical Language Understanding Evaluation Benchmarkcited by 99
  30. 2024AI Hospital: Benchmarking Large Language Models in a Multi-agent Medical Interaction Simulatorcited by 99
  31. 2024BioMedLM: A 2.7B Parameter Language Model Trained On Biomedical Textcited by 85
  32. 2022BioRED: A Rich Biomedical Relation Extraction Datasetcited by 84
  33. 2023Diagnostic Reasoning Prompts Reveal the Potential for Large Language Model Interpretability in Medicinecited by 83
  34. 2021MS2: Multi-Document Summarization of Medical Studiescited by 80
  35. 2024Sufficient Context: A New Lens on Retrieval Augmented Generation Systemscited by 69
  36. 2024A Framework for Human Evaluation of Large Language Models in Healthcare Derived from Literature Reviewcited by 65
  37. 2024MedCalc-Bench: Evaluating Large Language Models for Medical Calculationscited by 64
  38. 2025MedHELM: Holistic Evaluation of Large Language Models for Medical Taskscited by 60

50 most cited of 78 All 78 (JSON)

Built on by

Not cited yet.