mlsanityData to 14 Oct 2026

Disentangling Reasoning and Knowledge in Medical Large Language Models

arXiv 2505.11462 ↗First posted 16 May 2025cs.CL cs.AIcited by 18

On the map: LLM reasoning → Clinical and biomedical applications

Abstract (arXiv)

Medical reasoning in large language models (LLMs) aims to emulate clinicians' diagnostic thinking, but current benchmarks such as MedQA-USMLE, MedMCQA, and PubMedQA often mix reasoning with factual recall. We address this by separating 11 biomedical QA benchmarks into reasoning- and knowledge-focused subsets using a PubMedBERT classifier that reaches 81 percent accuracy, comparable to human performance. Our analysis shows that only 32.8 percent of questions require complex reasoning. We evaluate biomedical models (HuatuoGPT-o1, MedReason, m1) and general-domain models (DeepSeek-R1, o4-mini, Qwen3), finding consistent gaps between knowledge and reasoning performance. For example, HuatuoGPT-o1 scores 56.9 on knowledge but only 44.8 on reasoning. In adversarial tests where models are misled with incorrect initial reasoning, biomedical models degrade sharply, while larger or RL-trained general models show more robustness. To address this, we train BioMed-R1 using fine-tuning and reinforcement learning on reasoning-heavy examples. It achieves the strongest performance among similarly sized models. Further gains may come from incorporating clinical case reports and training with adversarial and backtracking scenarios.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Placed in Clinical and biomedical applications, one of 17 directions in LLM reasoning. Builds on 26 of its 50 references that are in the map. First posted on arXiv 16 May 2025.

Builds on

26 of 50 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025MedReason: Eliciting Factual Medical Reasoning Steps in LLMs via Knowledge Graphs1 Apr
  2. 2025m1: Unleash the Potential of Test-Time Scaling for Medical Reasoning with Large Language Models1 Apr
  3. 2024HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMs25 Dec

Other references · 23

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  3. 2024Qwen2.5 Technical Reportcited by 4,942
  4. 2024GPT-4o System Cardcited by 4,588
  5. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  6. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  7. 2024OpenAI o1 System Cardcited by 1,244
  8. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,210
  9. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 672
  10. 2025Humanity's Last Examcited by 491
  11. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 426
  12. 2025R1-Onevision: Advancing Generalized Multimodal Reasoning through Cross-Modal Formalizationcited by 370
11 more
  1. 2022Generating Sequences by Learning to Self-Correctcited by 277
  2. 2025MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understandingcited by 224
  3. 2025SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Language Modelscited by 212
  4. 2024Evaluation of OpenAI o1: Opportunities and Challenges of AGIcited by 127
  5. 2025OpenVLThinker: Complex Vision-Language Reasoning via Iterative SFT-RL Cyclescited by 124
  6. 2024Benchmarking Large Language Models on Answering and Explaining Challenging Medical Questionscited by 109
  7. 2019HEAD-QA: A Healthcare Dataset for Complex Reasoningcited by 101
  8. 2023Navigating the Grey Area: How Expressions of Uncertainty and Overconfidence Affect Language Modelscited by 91
  9. 2024Disentangling Memory and Reasoning Ability in Large Language Modelscited by 44
  10. 2024Backtracking Improves Generation Safetycited by 32
  11. 2025Lessons From Red Teaming 100 Generative AI Productscited by 23

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 10; 2026: 8.

18 papers.

Sort:

By area:

  1. 2025MedAgentGym: A Scalable Agentic Training Environment for Code-Centric Reasoning in Biomedical Data Sciencecited by 23
  2. 2025Medical Reasoning in the Era of LLMs: A Systematic Review of Enhancement Techniques and Applicationscited by 10
  3. 2025Bridging the Gap in Ophthalmic AI: MM-Retinal-Reason Dataset and OphthaReason Model toward Dynamic Multimodal Reasoningcited by 6
  4. 2026Evo-MedAgent: Beyond One-Shot Diagnosis with Agents That Remember, Reflect, and Improvecited by 6
  5. 2025RCP-Merging: Merging Long Chain-of-Thought Models with Domain-Specific Models by Considering Reasoning Capability as Priorcited by 5
  6. 2025Demystifying Scientific Problem-Solving in LLMs by Probing Knowledge and Reasoningcited by 4
  7. 2025Training and Evaluation of Guideline-Based Medical Reasoning in LLMscited by 4
  8. 2026Medical Reasoning with Large Language Models: A Survey and MR-Benchcited by 4
  9. 2025Rethinking Test-Time Scaling for Medical AI: Model and Task-Aware Strategies for LLMs and VLMscited by 3
  10. 2025How Far Have Medical Vision-Language Models Come? A Comprehensive Benchmarking Studycited by 3
  11. 2025Towards Efficient Medical Reasoning with Minimal Fine-Tuning Datacited by 3
  12. 2025Multilingual Medical Reasoning for Question Answering with Large Language Modelscited by 2
6 more
  1. 2026Quark Medical Alignment: A Holistic Multi-Dimensional Alignment and Collaborative Optimization Paradigmcited by 2
  2. 2026ReasonAny: Incorporating Reasoning Capability to Any Model via Simple and Effective Model Mergingcited by 1
  3. 2026Process Reward Agents for Steering Knowledge-Intensive Reasoningcited by 1
  4. 2026Eliciting Medical Reasoning with Knowledge-enhanced Data Synthesis: A Semi-Supervised Reinforcement Learning Approachcited by 1
  5. 2026ActiveMedAgent: Cost-Aware Trajectory Learning for Multimodal Medical Diagnosiscited by 0
  6. 2026MedEvo: Medical Reasoning Evolution through Knowledge Navigationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Automating Expert-Level Medical Reasoning Evaluation of Large Language Models
  2. 2026Medmarks: A Comprehensive Open-Source LLM Benchmark Suite for Medical Tasks
  3. 2026Advances in LLM Reasoning Enable Flexibility in Clinical Problem-Solving
  4. 2024Critique of Impure Reason: Unveiling the reasoning behaviour of medical Large Language Models
  5. 2025Fact or Guesswork? Evaluating Large Language Models' Medical Knowledge with Structured One-Hop Judgments
  6. 2024A Survey for Large Language Models in Biomedicine
  7. 2025MedOmni-45deg: A Safety-Performance Benchmark for Reasoning-Oriented LLMs in Medicine
  8. 2025Quantifying the Reasoning Abilities of LLMs on Real-world Clinical Cases