mlsanityData to 7 Oct 2026

Beyond MedQA: Towards Real-world Clinical Decision Making in the Era of LLMs

arXiv 2510.20001 ↗First posted 22 Oct 2025cs.CL cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) show promise for clinical use. They are often evaluated using datasets such as MedQA. However, Many medical datasets, such as MedQA, rely on simplified Question-Answering (QA) that underrepresents real-world clinical decision-making. Based on this, we propose a unifying paradigm that characterizes clinical decision-making tasks along two dimensions: Clinical Backgrounds and Clinical Questions. As the background and questions approach the real clinical environment, the difficulty increases. We summarize the settings of existing datasets and benchmarks along two dimensions. Then we review methods to address clinical decision-making, including training-time and test-time techniques, and summarize when they help. Next, we extend evaluation beyond accuracy to include efficiency, explainability. Finally, we highlight open challenges. Our paradigm clarifies assumptions, standardizes comparisons, and guides the development of clinically meaningful LLMs.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 37 of its 62 references that are in the map. First posted on arXiv 22 Oct 2025.

Builds on

37 of 62 references.

References · 37

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  6. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  7. 2024A Survey on LLM-as-a-Judgecited by 1,384
  8. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  9. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  10. 2024Benchmarking Retrieval-Augmented Generation for Medicinecited by 326
  11. 2024Capabilities of Gemini Models in Medicinecited by 231
  12. 2023How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Compositioncited by 222
25 more
  1. 2024AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environmentscited by 215
  2. 2023PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documentscited by 212
  3. 2025MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understandingcited by 212
  4. 2024MediQ: Question-Asking LLMs and a Benchmark for Reliable Interactive Clinical Reasoningcited by 145
  5. 2024AI Hospital: Benchmarking Large Language Models in a Multi-agent Medical Interaction Simulatorcited by 99
  6. 2023DISC-MedLLM: Bridging General Large Language Models and Real-World Medical Consultationcited by 89
  7. 2024MedCalc-Bench: Evaluating Large Language Models for Medical Calculationscited by 64
  8. 2024Reasoning-Enhanced Healthcare Predictions with Knowledge Graph Community Retrievalcited by 45
  9. 2025Evolving Language Models without Labels: Majority Drives Selection, Novelty Promotes Variationcited by 44
  10. 2024CoD, Towards an Interpretable Medical Agent using Chain of Diagnosiscited by 43
  11. 2024LongHealth: A Question Answering Benchmark with Long Clinical Documentscited by 33
  12. 2024Few shot chain-of-thought driven reasoning to prompt LLMs for open ended medical question answeringcited by 31
  13. 2024ClinicalBench: Can LLMs Beat Traditional ML Models in Clinical Prediction?cited by 31
  14. 2024ClinicalLab: Aligning Agents for Multi-Departmental Clinical Diagnostics in the Real Worldcited by 28
  15. 2025Cats Confuse Reasoning LLM: Query Agnostic Adversarial Triggers for Reasoning Modelscited by 23
  16. 2024Enhancing Diagnostic Accuracy through Multi-Agent Conversations: Using Large Language Models to Mitigate Cognitive Biascited by 20
  17. 2023Large Language Models Perform Diagnostic Reasoningcited by 18
  18. 2025ProMed: Shapley Information Gain Guided Reinforcement Learning for Proactive Medical LLMscited by 12
  19. 2025Addressing Benchmarking Gaps in Large Language Models for Health and Medicine with Dynamic Red-Teamingcited by 10
  20. 2025MedEthicEval: Evaluating Large Language Models Based on Chinese Medical Ethicscited by 8
  21. 2025AOR: Anatomical Ontology-Guided Reasoning for Medical Large Multimodal Model in Chest X-Ray Interpretationcited by 8
  22. 2025MedEthicsQA: A Comprehensive Question Answering Benchmark for Medical Ethics Evaluation of LLMscited by 8
  23. 2025MedTVT-R1: A Multimodal LLM Empowering Medical Reasoning and Diagnosiscited by 7
  24. 2024BP4ER: Bootstrap Prompting for Explicit Reasoning in Medical Dialogue Generationcited by 3
  25. 2025PathCoT: Chain-of-Thought Prompting for Zero-shot Pathology Visual Reasoningcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026EHRBench: An Automated and Reliable EHR-based Benchmark for Clinical Decision Making with LLMscited by 2