mlsanityData to 14 Oct 2026

Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine

arXiv 2311.16452 ↗First posted 28 Nov 2023cs.CLcited by 329

Not on the map. Why?

Cited by 51 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Generalist foundation models such as GPT-4 have displayed surprising capabilities in a wide variety of domains and tasks. Yet, there is a prevalent assumption that they cannot match specialist capabilities of fine-tuned models. For example, most explorations to date on medical competency benchmarks have leveraged domain-specific training, as exemplified by efforts on BioGPT and Med-PaLM. We build on a prior study of GPT-4's capabilities on medical challenge benchmarks in the absence of special training. Rather than using simple prompting to highlight the model's out-of-the-box capabilities, we perform a systematic exploration of prompt engineering. We find that prompting innovation can unlock deeper specialist capabilities and show that GPT-4 easily tops prior leading results for medical benchmarks. The prompting methods we explore are general purpose, and make no specific use of domain expertise, removing the need for expert-curated content. Our experimental design carefully controls for overfitting during the prompt engineering process. We introduce Medprompt, based on a composition of several prompting strategies. With Medprompt, GPT-4 achieves state-of-the-art results on all nine of the benchmark datasets in the MultiMedQA suite. The method outperforms leading specialist models such as Med-PaLM 2 by a significant margin with an order of magnitude fewer calls to the model. Steering GPT-4 with Medprompt achieves a 27% reduction in error rate on the MedQA dataset over the best methods to date achieved with specialist models and surpasses a score of 90% for the first time. Beyond medical problems, we show the power of Medprompt to generalize to other domains and provide evidence for the broad applicability of the approach via studies of the strategy on exams in electrical engineering, machine learning, philosophy, accounting, law, nursing, and clinical psychology.

Key facts. Cited by 329 papers in the map (data through 14 Oct 2026). Builds on 28 of its 38 references that are in the map. First posted on arXiv 28 Nov 2023.

Builds on

28 of 38 references.

References · 28

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  8. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  9. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  10. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  11. 2022Emergent Abilities of Large Language Modelscited by 2,512
  12. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
16 more
  1. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,356
  2. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  3. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  4. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 1,009
  5. 2023Large Language Models as Optimizerscited by 864
  6. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 638
  7. 2023Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolutioncited by 551
  8. 2022MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answeringcited by 543
  9. 2022BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Miningcited by 538
  10. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 486
  11. 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 263
  12. 2022LinkBERT: Pretraining Language Models with Document Linkscited by 211
  13. 2022Deep Bidirectional Language-Knowledge Graph Pretrainingcited by 156
  14. 2020Look at the First Sentence: Position Bias in Question Answeringcited by 84
  15. 2021Assessing the Fairness of AI Systems: AI Practitioners' Processes, Challenges, and Needs for Supportcited by 63
  16. 2023Self-Verification Improves Few-Shot Clinical Information Extractioncited by 52

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 168; 2025: 96; 2026: 60.

329 papers.

Sort:

By area:

  1. 2024GPT-4o System Cardcited by 4,588
  2. 2024AFlow: Automating Agentic Workflow Generationcited by 432
  3. 2024Benchmarking Retrieval-Augmented Generation for Medicinecited by 331
  4. 2024TextGrad: Automatic "Differentiation" via Textcited by 273
  5. 2024HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMscited by 265
  6. 2024Capabilities of Gemini Models in Medicinecited by 237
  7. 2024AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environmentscited by 231
  8. 2024Agent Hospital: A Simulacrum of Hospital with Evolvable Medical Agentscited by 218
  9. 2024MDAgents: An Adaptive Collaboration of LLMs for Medical Decision-Makingcited by 163
  10. 2024Exploring scalable medical image encoders beyond text supervisioncited by 132
  11. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  12. 2024Empowering Biomedical Discovery with AI Agentscited by 118
38 more
  1. 2024Benchmarking Large Language Models on Answering and Explaining Challenging Medical Questionscited by 109
  2. 2024MCS-SQL: Leveraging Multiple Prompts and Multiple-Choice Selection For Text-to-SQL Generationcited by 105
  3. 2024Code Generation with AlphaCodium: From Prompt Engineering to Flow Engineeringcited by 103
  4. 2024Towards Building Multilingual Language Model for Medicinecited by 92
  5. 2024What Did I Do Wrong? Quantifying LLMs' Sensitivity and Consistency to Prompt Engineeringcited by 90
  6. 2024Medical Graph RAG: Towards Safe Medical Large Language Model via Graph Retrieval-Augmented Generationcited by 90
  7. 2024Med42-v2: A Suite of Clinical LLMscited by 90
  8. 2024BioMedLM: A 2.7B Parameter Language Model Trained On Biomedical Textcited by 86
  9. 2024Towards Conversational Diagnostic AIcited by 79
  10. 2024Foundation Model for Advancing Healthcare: Challenges, Opportunities, and Future Directionscited by 77
  11. 2024LLM2LLM: Boosting LLMs with Novel Iterative Data Enhancementcited by 70
  12. 2024MedCalc-Bench: Evaluating Large Language Models for Medical Calculationscited by 70
  13. 2024UltraMedical: Building Specialized Generalists in Biomedicinecited by 67
  14. 2024Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wiselycited by 66
  15. 2026MIRAGE: The Illusion of Visual Understandingcited by 65
  16. 2025Sequential Diagnosis with Language Modelscited by 60
  17. 2025ScoreFlow: Mastering LLM Agent Workflows via Score-based Preference Optimizationcited by 56
  18. 2024HARGPT: Are LLMs Zero-Shot Human Activity Recognizers?cited by 54
  19. 2024Improving Retrieval-Augmented Generation in Medicine with Iterative Follow-up Questionscited by 54
  20. 2025SeRL: Self-Play Reinforcement Learning for Large Language Models with Limited Datacited by 51
  21. 2024Designing Heterogeneous LLM Agents for Financial Sentiment Analysiscited by 48
  22. 2023The Persuasive Power of Large Language Modelscited by 47
  23. 2024MedImageInsight: An Open-Source Embedding Model for General Domain Medical Imagingcited by 46
  24. 2024Retrieval Augmented Time Series Forecastingcited by 46
  25. 2024RareBench: Can LLMs Serve as Rare Diseases Specialists?cited by 45
  26. 2024A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicinecited by 45
  27. 2024From LLM to Conversational Agent: A Memory Enhanced Architecture with Fine-Tuning of Large Language Modelscited by 43
  28. 2025Med-RLVR: Emerging Medical Reasoning from a 3B base model via reinforcement Learningcited by 40
  29. 2025MedicalAgentsBench for Complex Medical Reasoning: Comparing Internalized Reasoning Models versus Externalized Agent-based Frameworkscited by 40
  30. 2025Self-Evolving Multi-Agent Simulations for Realistic Clinical Interactionscited by 40
  31. 2024In-context learning enables multimodal large language models to classify cancer pathology imagescited by 38
  32. 2024Addressing cognitive bias in medical language modelscited by 36
  33. 2024EHRNoteQA: An LLM Benchmark for Real-World Clinical Practice Using Discharge Summariescited by 35
  34. 2024A Survey on Medical Large Language Models: Technology, Application, Trustworthiness, and Future Directionscited by 34
  35. 2024How Susceptible are LLMs to Influence in Prompts?cited by 33
  36. 2024A Preliminary Study of o1 in Medicine: Are We Closer to an AI Doctor?cited by 33
  37. 2024Hidden flaws behind expert-level accuracy of multimodal GPT-4 vision in medicinecited by 31
  38. 2025X-Reasoner: Towards Generalizable Reasoning Across Modalities and Domainscited by 31

50 most cited of 329 All 329 (JSON)