mlsanityData to 7 Oct 2026

MedCalc-Bench: Evaluating Large Language Models for Medical Calculations

arXiv 2406.12036 ↗First posted 17 Jun 2024cs.CL cs.AIcited by 64

Not on the map. Why?

Abstract (arXiv)

As opposed to evaluating computation and logic-based reasoning, current benchmarks for evaluating large language models (LLMs) in medicine are primarily focused on question-answering involving domain knowledge and descriptive reasoning. While such qualitative capabilities are vital to medical diagnosis, in real-world scenarios, doctors frequently use clinical calculators that follow quantitative equations and rule-based reasoning paradigms for evidence-based decision support. To this end, we propose MedCalc-Bench, a first-of-its-kind dataset focused on evaluating the medical calculation capability of LLMs. MedCalc-Bench contains an evaluation set of over 1000 manually reviewed instances from 55 different medical calculation tasks. Each instance in MedCalc-Bench consists of a patient note, a question requesting to compute a specific medical value, a ground truth answer, and a step-by-step explanation showing how the answer is obtained. While our evaluation results show the potential of LLMs in this area, none of them are effective enough for clinical settings. Common issues include extracting the incorrect entities, not using the correct equation or rules for a calculation task, or incorrectly performing the arithmetic for the computation. We hope our study highlights the quantitative knowledge and reasoning gaps in LLMs within medical settings, encouraging future improvements of LLMs for various clinical calculation tasks.

Key facts. Cited by 64 papers in the map (data through 7 Oct 2026). Builds on 42 of its 75 references that are in the map. First posted on arXiv 17 Jun 2024.

Builds on

42 of 75 references.

References · 42

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2020Scaling Laws for Neural Language Modelscited by 6,411
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  10. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  11. 2023Mistral 7Bcited by 3,065
  12. 2022Training Compute-Optimal Large Language Modelscited by 2,927
30 more
  1. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  2. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  3. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  4. 2024Mixtral of Expertscited by 1,785
  5. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  6. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  7. 2023PaLM 2 Technical Reportcited by 1,154
  8. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  9. 2022PAL: Program-aided Language Modelscited by 635
  10. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 630
  11. 2023ChemCrow: Augmenting large-language models with chemistry toolscited by 609
  12. 2022MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answeringcited by 525
  13. 2023Cognitive Architectures for Language Agentscited by 422
  14. 2023Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicinecited by 319
  15. 2023Tool Learning with Foundation Modelscited by 302
  16. 2023ToolQA: A Dataset for LLM Question Answering with External Toolscited by 273
  17. 2023MEDITRON-70B: Scaling Medical Pretraining for Large Language Modelscited by 272
  18. 2023Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarizationcited by 244
  19. 2024Capabilities of Gemini Models in Medicinecited by 231
  20. 2022Can large language models reason about medical questions?cited by 194
  21. 2023Opportunities and Challenges for ChatGPT and Large Language Models in Biomedicine and Healthcited by 118
  22. 2023GeneGPT: Augmenting Large Language Models with Domain Tools for Improved Access to Biomedical Informationcited by 115
  23. 2023Matching Patients to Clinical Trials with Large Language Modelscited by 68
  24. 2023Summarizing, Simplifying, and Synthesizing Medical Evidence Using GPT-3 (with Varying Success)cited by 40
  25. 2022BigBIO: A Framework for Data-Centric Biomedical Natural Language Processingcited by 36
  26. 2024Zero-Shot Clinical Trial Patient Matching with LLMscited by 35
  27. 2024AgentMD: Empowering Language Agents for Risk Prediction with Large-Scale Clinical Tool Learningcited by 33
  28. 2023Structured Chemistry Reasoning with Large Language Modelscited by 27
  29. 2023Scaling Clinical Trial Matching Using Large Language Models: A Case Study in Oncologycited by 24
  30. 2024Team IELAB at TREC Clinical Trial Track 2023: Enhancing Clinical Trial Retrieval with Neural Rankers and Large Language Modelscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 25; 2026: 34.

64 papers.

Sort:

By area:

  1. 2026Skill Retrieval Augmentation for Agentic AIcited by 41
  2. 2024A Preliminary Study of o1 in Medicine: Are We Closer to an AI Doctor?cited by 33
  3. 2025LLMEval-Med: A Real-world Clinical Benchmark for Medical LLMs with Physician Validationcited by 29
  4. 2024MEDIC: Comprehensive Evaluation of Leading Indicators for LLM Safety and Utility in Clinical Applicationscited by 25
  5. 2025MedS3: Towards Medical Slow Thinking with Self-Evolved Soft Dual-sided Process Supervisioncited by 23
  6. 2025SFT Doesn't Always Hurt General Capabilities: Revisiting Domain-Specific Fine-Tuning in LLMscited by 23
  7. 2025MedAgentGym: A Scalable Agentic Training Environment for Code-Centric Reasoning in Biomedical Data Sciencecited by 22
  8. 2026HealthBench Professional: Evaluating Large Language Models on Real Clinician Chatscited by 22
  9. 2024ReflecTool: Towards Reflection-Aware Tool-Augmented Clinical Agentscited by 20
  10. 2025Loong: Synthesize Long Chain-of-Thoughts at Scale through Verifierscited by 14
  11. 2025From Scores to Steps: Diagnosing and Improving LLM Performance in Evidence-Based Medical Calculationscited by 11
  12. 2025LLM-based HSE Compliance Assessment: Benchmark, Performance, and Advancementscited by 8
38 more
  1. 2025Why Chain of Thought Fails in Clinical Text Understandingcited by 8
  2. 2026Detecting LLM-Generated Text with Performance Guaranteescited by 8
  3. 2025AECBench: A Hierarchical Benchmark for Knowledge Evaluation of Large Language Models in the AEC Fieldcited by 6
  4. 2025QuarkMed Medical Foundation Model Technical Reportcited by 5
  5. 2026Breaking the Chain: A Causal Analysis of LLM Faithfulness to Intermediate Structurescited by 5
  6. 2025Language Models Do Not Embed Numbers Continuouslycited by 4
  7. 2025CP-Env: Evaluating Large Language Models on Clinical Pathways in a Controllable Hospital Environmentcited by 4
  8. 2026Do Instruction-Tuned Models Always Perform Better Than Base Models? Evidence from Math and Domain-Shifted Benchmarkscited by 4
  9. 2026Right Family, Wrong Skill: Evaluating Risk Exposure in Agent Skill Retrievalcited by 4
  10. 2024Entry-level guide to the use of large language models for medical researchcited by 3
  11. 2025RiskAgent: Synergizing Language Models with Validated Tools for Evidence-Based Risk Predictioncited by 3
  12. 2025GuessArena: Guess Who I Am? A Self-Adaptive Framework for Evaluating LLMs in Domain-Specific Knowledge and Reasoningcited by 3
  13. 2025Structured Prompts Improve Evaluation of Language Modelscited by 3
  14. 2026MedCalc-Bench Doesn't Measure What You Think: A Benchmark Audit and the Case for Open-Book Evaluationcited by 3
  15. 2026MedCUA-Bench: A Screenshot-Only Benchmark for Clinical Computer-Use Agentscited by 3
  16. 2025Semi-structured LLM Reasoners Can Be Rigorously Auditedcited by 2
  17. 2025Domain Specific Benchmarks for Evaluating Multimodal Large Language Modelscited by 2
  18. 2025DM-Bench: Benchmarking LLMs for Personalized Decision Making in Diabetes Managementcited by 2
  19. 2025Scalable Stewardship of an LLM-Assisted Clinical Benchmark with Physician Oversightcited by 2
  20. 2026MedMCP-Calc: Benchmarking LLMs for Realistic Medical Calculator Scenarios via MCP Integrationcited by 2
  21. 2026Can LLM Agents Generate Real-World Evidence? Evaluating Observational Studies in Medical Databasescited by 2
  22. 2026Does Accuracy Equal Evidence? Reasoning Faithfulness under KV Cache Compressioncited by 2
  23. 2026SkillApt: Learning When to Activate Agent Skills from Counterfactual Evidencecited by 2
  24. 2024Humans and Large Language Models in Clinical Decision Support: A Study with Medical Calculatorscited by 1
  25. 2025OphthBench: A Comprehensive Benchmark for Evaluating Large Language Models in Chinese Ophthalmologycited by 1
  26. 2025HealthBranches: Synthesizing Clinically-Grounded Question Answering Datasets via Decision Pathwayscited by 1
  27. 2025Beyond MedQA: Towards Real-world Clinical Decision Making in the Era of LLMscited by 1
  28. 2026Stabilizing LLM Supervised Fine-Tuning via Explicit Distributional Controlcited by 1
  29. 2026DDX-TRACE: A Benchmark for Medical Diagnostic Trajectories in VLMscited by 1
  30. 2026Adaptive Multi-Resolution Procedural Knowledge Compression for Large Language Modelscited by 1
  31. 2026The Capability Frontier: Benchmarks Miss 82% of Model Performancecited by 1
  32. 2026The Path to Self-Evolving Clinical Systems: Scaling Medical Agents from Assistance to Autonomycited by 1
  33. 2025CMQCIC-Bench: A Chinese Benchmark for Evaluating Large Language Models in Medical Quality Control Indicator Calculationcited by 0
  34. 2025Evaluating Medical LLMs by Levels of Autonomy: A Survey Moving from Benchmarks to Applicationscited by 0
  35. 2025MedCalc-Eval and MedCalc-Env: Advancing Medical Calculation Capabilities of Large Language Modelscited by 0
  36. 2026ADRD-Bench: A Preliminary LLM Benchmark for Alzheimer's Disease and Related Dementiascited by 0
  37. 2026Perfecting Human-AI Interaction at Clinical Scale. Turning Production Signals into Safer, More Human Conversationscited by 0
  38. 2026Learning to Construct Practical Agentic Systemscited by 0

50 most cited of 64 All 64 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Automating Expert-Level Medical Reasoning Evaluation of Large Language Models
  2. 2025Quantifying the Reasoning Abilities of LLMs on Real-world Clinical Cases
  3. 2026Medical Reasoning with Large Language Models: A Survey and MR-Bench
  4. 2024MedExpQA: Multilingual Benchmarking of Large Language Models for Medical Question Answering
  5. 2026How Robust Are Large Language Models for Clinical Numeracy? An Empirical Study on Numerical Reasoning Abilities in Clinical Contexts
  6. 2026Medmarks: A Comprehensive Open-Source LLM Benchmark Suite for Medical Tasks
  7. 2025MedCaseReasoning: Evaluating and learning diagnostic reasoning from clinical case reports
  8. 2025MedOmni-45deg: A Safety-Performance Benchmark for Reasoning-Oriented LLMs in Medicine