mlsanityData to 7 Oct 2026

From Medprompt to o1: Exploration of Run-Time Strategies for Medical Challenge Problems and Beyond

arXiv 2411.03590 ↗First posted 6 Nov 2024cs.CLcited by 24

Not on the map. Why?

Abstract (arXiv)

Run-time steering strategies like Medprompt are valuable for guiding large language models (LLMs) to top performance on challenging tasks. Medprompt demonstrates that a general LLM can be focused to deliver state-of-the-art performance on specialized domains like medicine by using a prompt to elicit a run-time strategy involving chain of thought reasoning and ensembling. OpenAI's o1-preview model represents a new paradigm, where a model is designed to do run-time reasoning before generating final responses. We seek to understand the behavior of o1-preview on a diverse set of medical challenge problem benchmarks. Following on the Medprompt study with GPT-4, we systematically evaluate the o1-preview model across various medical benchmarks. Notably, even without prompting techniques, o1-preview largely outperforms the GPT-4 series with Medprompt. We further systematically study the efficacy of classic prompt engineering strategies, as represented by Medprompt, within the new paradigm of reasoning models. We found that few-shot prompting hinders o1's performance, suggesting that in-context learning may no longer be an effective steering approach for reasoning-native models. While ensembling remains viable, it is resource-intensive and requires careful cost-performance optimization. Our cost and accuracy analysis across run-time strategies reveals a Pareto frontier, with GPT-4o representing a more affordable option and o1-preview achieving state-of-the-art performance at higher cost. Although o1-preview offers top performance, GPT-4o with steering strategies like Medprompt retains value in specific contexts. Moreover, we note that the o1-preview model has reached near-saturation on many existing medical benchmarks, underscoring the need for new, challenging benchmarks. We close with reflections on general directions for inference-time computation with LLMs.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 40 of its 52 references that are in the map. First posted on arXiv 6 Nov 2024.

Builds on

40 of 52 references.

References · 40

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  9. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  10. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  11. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  12. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
28 more
  1. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  2. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  3. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  4. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  5. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  6. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 996
  7. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  8. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 630
  9. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  10. 2022BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Miningcited by 531
  11. 2022MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answeringcited by 525
  12. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 419
  13. 2023Towards Expert-Level Medical Question Answering with Large Language Modelscited by 382
  14. 2023Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicinecited by 319
  15. 2024Quiet-STaR: Language Models Can Teach Themselves to Think Before Speakingcited by 292
  16. 2024Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Modelscited by 282
  17. 2024Capabilities of Gemini Models in Medicinecited by 231
  18. 2024Many-Shot In-Context Learningcited by 203
  19. 2024Let's Think Dot by Dot: Hidden Computation in Transformer Language Modelscited by 186
  20. 2024BioMedLM: A 2.7B Parameter Language Model Trained On Biomedical Textcited by 85
  21. 2020Look at the First Sentence: Position Bias in Question Answeringcited by 83
  22. 2023Evaluating GPT-4 and ChatGPT on Japanese Medical Licensing Examinationscited by 63
  23. 2024Reasoning Aware Self-Consistency: Leveraging Reasoning Paths for Efficient LLM Samplingcited by 63
  24. 2024Make Every Penny Count: Difficulty-Adaptive Self-Consistency for Cost-Efficient Reasoningcited by 61
  25. 2024Thinking LLMs: General Instruction Following with Thought Generationcited by 45
  26. 2013Reasoning About Beliefs and Actions Under Computational Resource Constraintscited by 18
  27. 2013A Bayesian Approach to Tackling Hard Computational Problemscited by 5
  28. 2021Ideal Partition of Resources for Metareasoningcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 17; 2026: 5.

24 papers.

Sort:

By area:

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  2. 2024HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMscited by 257
  3. 2025Sequential Diagnosis with Language Modelscited by 53
  4. 2025Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworkscited by 37
  5. 2025X-Reasoner: Towards Generalizable Reasoning Across Modalities and Domainscited by 30
  6. 2024Superhuman performance of a large language model on the reasoning tasks of a physiciancited by 28
  7. 2025MedS3: Towards Medical Slow Thinking with Self-Evolved Soft Dual-sided Process Supervisioncited by 23
  8. 2025A Modular Approach for Clinical SLMs Driven by Synthetic Data with Pre-Instruction Tuning, Model Merging, and Clinical-Tasks Alignmentcited by 20
  9. 2025DiagnosisArena: Benchmarking Diagnostic Reasoning for Large Language Modelscited by 19
  10. 2025On the Effect of Sampling Diversity in Scaling LLM Inferencecited by 18
  11. 2025From Scores to Steps: Diagnosing and Improving LLM Performance in Evidence-Based Medical Calculationscited by 11
  12. 2025DISC: Dynamic Decomposition Improves LLM Inference Scalingcited by 9
12 more
  1. 2025Empowering Healthcare Practitioners with Language Models: Structuring Speech Transcripts in Two Real-World Clinical Applicationscited by 7
  2. 2025Stabilizing Reasoning in Medical LLMs with Continued Pretraining and Reasoning Preference Optimizationcited by 4
  3. 2025Structured Prompts Improve Evaluation of Language Modelscited by 3
  4. 2025A Japanese Language Model and Three New Evaluation Benchmarks for Pharmaceutical NLPcited by 2
  5. 2025RephQA: Evaluating Readability of Large Language Models in Public Health Question Answeringcited by 2
  6. 2025Benchmarking the Thinking Mode of Multimodal Large Language Models in Clinical Taskscited by 2
  7. 2026Note2Chat: Improving LLMs for Multi-Turn Clinical History Taking Using Medical Notescited by 1
  8. 2026How people use Copilot for Healthcited by 1
  9. 2025Iterative Tree Analysis for Medical Criticscited by 0
  10. 2026Reinforcement Learning for Evidence-Seeking Diagnostic Reasoning with Large Language Modelscited by 0
  11. 2026A collective capability boundary in frontier large language models on guideline-conformant and case-specific oncology decision-makingcited by 0
  12. 2026Synthetic Hospital: An Open, Verifiable, Physician-Validated Longitudinal EHR Benchmarkcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025AutoMedPrompt: A New Framework for Optimizing LLM Medical Prompts Using Textual Gradients
  2. 2024A Preliminary Study of o1 in Medicine: Are We Closer to an AI Doctor?
  3. 2025O1 Replication Journey -- Part 3: Inference-time Scaling for Medical Reasoning
  4. 2026When Chain-of-Thought Backfires: Evaluating Prompt Sensitivity in Medical Language Models
  5. 2023An Empirical Evaluation of Prompting Strategies for Large Language Models in Zero-Shot Clinical Natural Language Processing
  6. 2025TAGS: A Test-Time Generalist-Specialist Framework with Retrieval-Augmented Reasoning and Verification
  7. 2025m1: Unleash the Potential of Test-Time Scaling for Medical Reasoning with Large Language Models
  8. 2025Instruction Tuning and CoT Prompting for Contextual Medical QA with LLMs