mlsanityData to 14 Oct 2026

Benchmarking Large Language Models on Answering and Explaining Challenging Medical Questions

arXiv 2402.18060 ↗First posted 28 Feb 2024cs.CLcited by 109

Not on the map. Why?

Cited by 28 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

LLMs have demonstrated impressive performance in answering medical questions, such as achieving passing scores on medical licensing examinations. However, medical board exams or general clinical questions do not capture the complexity of realistic clinical cases. Moreover, the lack of reference explanations means we cannot easily evaluate the reasoning of model decisions, a crucial component of supporting doctors in making complex medical decisions. To address these challenges, we construct two new datasets: JAMA Clinical Challenge and Medbullets. Datasets and code are available at https://github.com/HanjieChen/ChallengeClinicalQA. JAMA Clinical Challenge consists of questions based on challenging clinical cases, while Medbullets comprises simulated clinical questions. Both datasets are structured as multiple-choice question-answering tasks, accompanied by expert-written explanations. We evaluate seven LLMs on the two datasets using various prompts. Experiments demonstrate that our datasets are harder than previous benchmarks. In-depth automatic and human evaluations of model-generated explanations provide insights into the promise and deficiency of LLMs for explainable medical QA.

Key facts. Cited by 109 papers in the map (data through 14 Oct 2026). Builds on 41 of its 82 references that are in the map. First posted on arXiv 28 Feb 2024.

Builds on

41 of 82 references.

References · 41

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  8. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  9. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  11. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  12. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
29 more
  1. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  2. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  3. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,356
  4. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  5. 2015Teaching Machines to Read and Comprehendcited by 1,049
  6. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 722
  7. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 638
  8. 2023The Falcon Series of Open Language Modelscited by 535
  9. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 348
  10. 2023Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicinecited by 329
  11. 2023MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Datacited by 286
  12. 2022Data Contamination: From Memorization to Exploitationcited by 236
  13. 2023HuatuoGPT, towards Taming Language Model to Be a Doctorcited by 214
  14. 2022Can large language models reason about medical questions?cited by 194
  15. 2021Meta-learning via Language Model In-context Tuningcited by 164
  16. 2023A Study of Generative Large Language Model for Medical Research and Healthcarecited by 163
  17. 2024When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboardscited by 137
  18. 2018emrQA: A Large Corpus for Question Answering on Electronic Medical Recordscited by 135
  19. 2019A Question-Entailment Approach to Question Answeringcited by 134
  20. 2023AlpaCare:Instruction-tuned Large Language Models for Medical Applicationcited by 90
  21. 2023ClinicalGPT: Large Language Models Finetuned with Diverse Medical Data and Comprehensive Evaluationcited by 88
  22. 2021Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural Language Generationcited by 60
  23. 2023Publicly Shareable Clinical Large Language Model Built on Synthetic Clinical Notescited by 51
  24. 2024Hidden flaws behind expert-level accuracy of multimodal GPT-4 vision in medicinecited by 31
  25. 2024K-QA: A Real-World Medical Q&A Benchmarkcited by 31
  26. 2020CliniQG4QA: Generating Diverse Questions for Domain Adaptation of Clinical Question Answeringcited by 26
  27. 2023MedEval: A Multi-Level, Multi-Task, and Multi-Domain Medical Benchmark for Language Model Evaluationcited by 19
  28. 2023ExplainCPE: A Free-text Explanation Benchmark of Chinese Pharmacist Examinationcited by 10
  29. 2023LongBoX: Evaluating Transformers on Long-Sequence Clinical Taskscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 15; 2025: 50; 2026: 44.

109 papers.

Sort:

By area:

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 384
  2. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 320
  3. 2025Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoningcited by 230
  4. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 223
  5. 2024MDAgents: An Adaptive Collaboration of LLMs for Medical Decision-Makingcited by 163
  6. 2025MedReason: Eliciting Factual Medical Reasoning Steps in LLMs via Knowledge Graphscited by 110
  7. 2025m1: Unleash the Potential of Test-Time Scaling for Medical Reasoning with Large Language Modelscited by 46
  8. 2025Baichuan-M1: Pushing the Medical Capability of Large Language Modelscited by 43
  9. 2025MedicalAgentsBench for Complex Medical Reasoning: Comparing Internalized Reasoning Models versus Externalized Agent-based Frameworkscited by 40
  10. 2025A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontierscited by 37
  11. 2025O1 Replication Journey -- Part 3: Inference-time Scaling for Medical Reasoningcited by 35
  12. 2024A Preliminary Study of o1 in Medicine: Are We Closer to an AI Doctor?cited by 33
38 more
  1. 2024ClinicalBench: Can LLMs Beat Traditional ML Models in Clinical Prediction?cited by 32
  2. 2024Small Language Models Learn Enhanced Reasoning Skills from Medical Textbookscited by 30
  3. 2024AfriMed-QA: A Pan-African, Multi-Specialty, Medical Question-Answering Benchmark Datasetcited by 26
  4. 2025Automating Expert-Level Medical Reasoning Evaluation of Large Language Modelscited by 22
  5. 2025DiagnosisArena: Benchmarking Diagnostic Reasoning for Large Language Modelscited by 21
  6. 2025Disentangling Reasoning and Knowledge in Medical Large Language Modelscited by 18
  7. 2025Large language models provide unsafe answers to patient-posed medical questionscited by 18
  8. 2025Beyond Log Likelihood: Probability-Based Objectives for Supervised Fine-Tuning across the Model Capability Continuumcited by 18
  9. 2024Medchain: Bridging the Gap Between LLM Agents and Clinical Practice with Interactive Sequencecited by 17
  10. 2025Citrus: Leveraging Expert Cognitive Pathways in a Medical Language Model for Advanced Medical Decision Supportcited by 16
  11. 2025Cancer-Myth: Evaluating Large Language Models on Patient Questions with False Presuppositionscited by 15
  12. 2025MedArabiQ: Benchmarking Large Language Models on Arabic Medical Taskscited by 15
  13. 2024DiReCT: Diagnostic Reasoning for Clinical Notes via Large Language Modelscited by 14
  14. 2025MultiFinBen: Benchmarking Large Language Models for Multilingual and Multimodal Financial Applicationcited by 14
  15. 2024Does Biomedical Training Lead to Better Medical Performance?cited by 13
  16. 2025Explainability in Practice: A Survey of Explainable NLP Across Various Domainscited by 11
  17. 2026Recursive Multi-Agent Systemscited by 10
  18. 2025Evaluating Reasoning Faithfulness in Medical Vision-Language Models using Multimodal Perturbationscited by 9
  19. 2026Search-Time Contamination in Deep Research Agents: Measuring Performance Inflation in Public Benchmark Evaluationcited by 9
  20. 2025Citrus-V: Advancing Medical Foundation Models with Unified Medical Image Grounding for Clinical Reasoningcited by 8
  21. 2026From Conflict to Consensus: Boosting Medical Reasoning via Multi-Round Agentic RAGcited by 7
  22. 2024Evaluating Large Language Models for Public Health Classification and Extraction Taskscited by 6
  23. 2025Susceptibility of Large Language Models to User-Driven Factors in Medical Queriescited by 6
  24. 2025Elicit and Enhance: Advancing Multimodal Reasoning in Medical Scenarioscited by 6
  25. 2026MedXIAOHE: A Comprehensive Recipe for Building Medical MLLMscited by 6
  26. 2026When Correct Beliefs Collapse: Epistemic Resilience of LLMs under Clinical Pressurecited by 6
  27. 2024Comprehensive and Practical Evaluation of Retrieval-Augmented Generation Systems for Medical Question Answeringcited by 5
  28. 2025MedScore: Generalizable Factuality Evaluation of Free-Form Medical Answers by Domain-adapted Claim Decomposition and Verificationcited by 5
  29. 2025MedPAIR: Measuring Physicians and AI Relevance Alignment in Medical Question Answeringcited by 5
  30. 2025QuarkMed Medical Foundation Model Technical Reportcited by 5
  31. 2024How Can We Diagnose and Treat Bias in Large Language Models for Clinical Decision-Making?cited by 4
  32. 2025Dialogue is Better Than Monologue: Instructing Medical LLMs via Strategical Conversationscited by 4
  33. 2025Explaining Large Language Models with gSMILEcited by 4
  34. 2025Putting on the Thinking Hats: A Survey on Chain of Thought Fine-tuning from the Perspective of Human Reasoning Mechanismcited by 4
  35. 2025CP-Env: Evaluating Large Language Models on Clinical Pathways in a Controllable Hospital Environmentcited by 4
  36. 2026ClinicalAgents: Multi-Agent Orchestration for Clinical Decision Making with Dual-Memorycited by 4
  37. 2026Benchmarking Multi-turn Medical Diagnosis: Hold, Lure, and Self-Correctioncited by 4
  38. 2026Medical thinking with multiple imagescited by 4

50 most cited of 109 All 109 (JSON)