mlsanityData to 7 Oct 2026

MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understanding

arXiv 2501.18362 ↗First posted 30 Jan 2025cs.AI cs.CL cs.CVcited by 212

Not on the map. Why?

Cited by 91 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce MedXpertQA, a highly challenging and comprehensive benchmark to evaluate expert-level medical knowledge and advanced reasoning. MedXpertQA includes 4,460 questions spanning 17 specialties and 11 body systems. It includes two subsets, Text for text evaluation and MM for multimodal evaluation. Notably, MM introduces expert-level exam questions with diverse images and rich clinical information, including patient records and examination results, setting it apart from traditional medical multimodal benchmarks with simple QA pairs generated from image captions. MedXpertQA applies rigorous filtering and augmentation to address the insufficient difficulty of existing benchmarks like MedQA, and incorporates specialty board questions to improve clinical relevance and comprehensiveness. We perform data synthesis to mitigate data leakage risk and conduct multiple rounds of expert reviews to ensure accuracy and reliability. We evaluate 18 leading models on benchmark. Moreover, medicine is deeply connected to real-world decision-making, providing a rich and representative setting for assessing reasoning abilities beyond mathematics and code. To this end, we develop a reasoning-oriented subset to facilitate the assessment of o1-like models. Code and data are available at: https://github.com/TsinghuaC3I/MedXpertQA

Key facts. Cited by 212 papers in the map (data through 7 Oct 2026). Builds on 32 of its 50 references that are in the map. First posted on arXiv 30 Jan 2025.

Builds on

32 of 50 references.

References · 32

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2024Qwen2.5 Technical Reportcited by 4,713
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  7. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  8. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  9. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  10. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  11. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  12. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
20 more
  1. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  2. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 996
  3. 2022MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answeringcited by 525
  4. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
  5. 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 467
  6. 2025Humanity's Last Examcited by 467
  7. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 444
  8. 2023PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answeringcited by 333
  9. 2024HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMscited by 257
  10. 2024Capabilities of Gemini Models in Medicinecited by 231
  11. 2023MedCPT: Contrastive Pre-trained Transformers with Large-scale PubMed Search Logs for Zero-shot Biomedical Information Retrievalcited by 182
  12. 2024OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLMcited by 182
  13. 2024Rho-1: Not All Tokens Are What You Needcited by 132
  14. 2024Benchmarking Benchmark Leakage in Large Language Modelscited by 102
  15. 2024GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AIcited by 96
  16. 2024UltraMedical: Building Specialized Generalists in Biomedicinecited by 67
  17. 2024A Comparative Study on Reasoning Patterns of OpenAI's o1 Modelcited by 62
  18. 2025O1 Replication Journey -- Part 3: Inference-time Scaling for Medical Reasoningcited by 34
  19. 2024A Preliminary Study of o1 in Medicine: Are We Closer to an AI Doctor?cited by 33
  20. 2024Weighted Brier Score -- an Overall Summary Measure for Risk Prediction Models with Clinical Utility Considerationcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 90; 2026: 122.

212 papers.

Sort:

By area:

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  2. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  3. 2025Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoningcited by 217
  4. 2026Gemma 4 Technical Reportcited by 187
  5. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  6. 2026Qwen3.5-Omni Technical Reportcited by 146
  7. 2025MedReason: Eliciting Factual Medical Reasoning Steps in LLMs via Knowledge Graphscited by 105
  8. 2026MIRAGE: The Illusion of Visual Understandingcited by 57
  9. 2025Baichuan-M2: Scaling Medical Capability with Large Verifier Systemcited by 55
  10. 2026MedGemma 1.5 Technical Reportcited by 43
  11. 2025m1: Unleash the Potential of Test-Time Scaling for Medical Reasoning with Large Language Modelscited by 42
  12. 2025MedicalAgentsBench for Complex Medical Reasoning: Comparing Internalized Reasoning Models versus Externalized Agent-based Frameworkscited by 38
38 more
  1. 2025Patho-R1: A Multimodal Reinforcement Learning-Based Pathology Expert Reasonercited by 38
  2. 2025Capabilities of GPT-5 on Multimodal Medical Reasoningcited by 35
  3. 2025MedCaseReasoning: Evaluating and learning diagnostic reasoning from clinical case reportscited by 34
  4. 2025Beyond Distillation: Pushing the Limits of Medical LLM Reasoning with Minimalist Rule-Based RLcited by 34
  5. 2025MedFrameQA: A Multi-Image Medical VQA Benchmark for Clinical Reasoningcited by 33
  6. 2025ChestX-Reasoner: Advancing Radiology Foundation Models with Reasoning through Step-by-Step Verificationcited by 30
  7. 2025X-Reasoner: Towards Generalizable Reasoning Across Modalities and Domainscited by 30
  8. 2025MedVLThinker: Simple Baselines for Multimodal Medical Reasoningcited by 26
  9. 2025Pillar-0: A New Frontier for Radiology Foundation Modelscited by 24
  10. 2025MedAgentGym: A Scalable Agentic Training Environment for Code-Centric Reasoning in Biomedical Data Sciencecited by 22
  11. 2026HealthBench Professional: Evaluating Large Language Models on Real Clinician Chatscited by 22
  12. 2025Automating Expert-Level Medical Reasoning Evaluation of Large Language Modelscited by 21
  13. 2025DiagnosisArena: Benchmarking Diagnostic Reasoning for Large Language Modelscited by 19
  14. 2025Disentangling Reasoning and Knowledge in Medical Large Language Modelscited by 17
  15. 2025Improving Medical Reasoning with Curriculum-Aware Reinforcement Learningcited by 16
  16. 2025Chiron-o1: Igniting Multimodal Large Language Models towards Generalizable Medical Reasoning via Mentor-Intern Collaborative Searchcited by 16
  17. 2025Beyond Log Likelihood: Probability-Based Objectives for Supervised Fine-Tuning across the Model Capability Continuumcited by 16
  18. 2025BioProBench: A Corpus and Benchmark for Biological Protocol Reasoning in Autonomous Sciencecited by 14
  19. 2025Knowledge or Reasoning? A Close Look at How LLMs Think Across Domainscited by 14
  20. 2025Med-CMR: A Fine-Grained Benchmark Integrating Visual Evidence and Clinical Logic for Medical Complex Multimodal Reasoningcited by 14
  21. 2026Counteraction-Aware Multi-Teacher On-Policy Distillation for General Capability Recovery with Domain Preservationcited by 14
  22. 2025EndoBench: A Comprehensive Evaluation of Multi-Modal Large Language Models for Endoscopy Analysiscited by 13
  23. 2025Patho-AgenticRAG: Towards Multimodal Agentic Retrieval-Augmented Generation for Pathology VLMs via Reinforcement Learningcited by 13
  24. 2025Human Cognitive Benchmarks Reveal Foundational Visual Gaps in MLLMscited by 12
  25. 2025AdsQA: Towards Advertisement Video Understandingcited by 12
  26. 2025Think Twice to See More: Iterative Visual Reasoning in Medical VLMscited by 12
  27. 2026MedSynapse-V: Bridging Visual Perception and Clinical Intuition via Latent Memory Evolutioncited by 12
  28. 2025SPINE: Token-Selective Test-Time Reinforcement Learning with Entropy-Band Regularizationcited by 11
  29. 2026Computer Environments Elicit General Agentic Intelligence in LLMscited by 11
  30. 2025A Comparison of DeepSeek and Other LLMscited by 10
  31. 2025Medical Reasoning in the Era of LLMs: A Systematic Review of Enhancement Techniques and Applicationscited by 10
  32. 2025OctoMed: Data Recipes for State-of-the-Art Multimodal Medical Reasoningcited by 10
  33. 2025Fleming-R1: Toward Expert-Level Medical Reasoning via Reinforcement Learningcited by 9
  34. 2025Evaluating Reasoning Faithfulness in Medical Vision-Language Models using Multimodal Perturbationscited by 9
  35. 2025Med-U1: Incentivizing Unified Medical Reasoning in LLMs via Large-scale Reinforcement Learningcited by 8
  36. 2025Citrus-V: Advancing Medical Foundation Models with Unified Medical Image Grounding for Clinical Reasoningcited by 8
  37. 2025MedMMV: A Controllable Multimodal Multi-Agent Framework for Reliable and Verifiable Clinical Reasoningcited by 8
  38. 2025Stabilizing Off-Policy Training for Long-Horizon LLM Agent via Turn-Level Importance Sampling and Clipping-Triggered Normalizationcited by 8

50 most cited of 212 All 212 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025MedBookVQA: A Systematic and Comprehensive Medical Benchmark Derived from Open-Access Book
  2. 2024MedExpQA: Multilingual Benchmarking of Large Language Models for Medical Question Answering
  3. 2024MedCalc-Bench: Evaluating Large Language Models for Medical Calculations
  4. 2025Quantifying the Reasoning Abilities of LLMs on Real-world Clinical Cases
  5. 2025MedAtlas: Evaluating LLMs for Multi-Round, Multi-Task Medical Reasoning Across Diverse Imaging Modalities and Clinical Text
  6. 2025MedOmni-45deg: A Safety-Performance Benchmark for Reasoning-Oriented LLMs in Medicine
  7. 2024RJUA-MedDQA: A Multimodal Benchmark for Medical Document Question Answering and Clinical Reasoning
  8. 2026MedReaMM: Evaluating Large Multimodal Models on Expert-Level Clinical Diagnostic Synthesis