# AutoMedPrompt: A New Framework for Optimizing LLM Medical Prompts Using Textual Gradients

arXiv: https://arxiv.org/abs/2502.15944 · HTML: https://mlsanity.com/paper/2502.15944/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 25 references that are in the map.
- First posted on arXiv 21 Feb 2025.

## Builds on (most cited first, 14 of 14)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines](https://mlsanity.com/paper/2310.03714/) (2023)
- [A Survey on In-context Learning](https://mlsanity.com/paper/2301.00234/) (2022)
- [Automatic Prompt Optimization with "Gradient Descent" and Beam Search](https://mlsanity.com/paper/2305.03495/) (2023)
- [MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering](https://mlsanity.com/paper/2203.14371/) (2022)
- [The Falcon Series of Open Language Models](https://mlsanity.com/paper/2311.16867/) (2023)
- [Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine](https://mlsanity.com/paper/2311.16452/) (2023)
- [Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs](https://mlsanity.com/paper/2406.11695/) (2024)
- [Combating Mode Collapse in GAN training: An Empirical Analysis using Hessian Eigenvalues](https://mlsanity.com/paper/2012.09673/) (2020)
- [An Information-Theoretic Perspective on Overfitting and Underfitting](https://mlsanity.com/paper/2010.06076/) (2020)
- [ELMTEX: Fine-Tuning Large Language Models for Structured Clinical Information Extraction. A Case Study on Clinical Reports](https://mlsanity.com/paper/2502.05638/) (2025)

## Built on by (most cited first, 7 of 7)

- [The Rise of Small Language Models in Healthcare: A Comprehensive Survey](https://mlsanity.com/paper/2504.17119/) (2025)
- [Reasoning LLMs in the Medical Domain: A Literature Survey](https://mlsanity.com/paper/2508.19097/) (2025)
- [DoctorRAG: Medical RAG Fusing Knowledge with Patient Analogy through Textual Gradients](https://mlsanity.com/paper/2505.19538/) (2025)
- [One Patient, Many Contexts: Scaling Medical AI with Contextual Intelligence](https://mlsanity.com/paper/2506.10157/) (2025)
- [Advances in Large Language Models for Medicine](https://mlsanity.com/paper/2509.18690/) (2025)
- [DeCode: Decoupling Content and Delivery for Medical QA](https://mlsanity.com/paper/2601.02123/) (2026)
- [LLM-Guided Evolution for Medical Decision Pipelines](https://mlsanity.com/paper/2606.07342/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have demonstrated increasingly sophisticated performance in medical and other fields of knowledge. Traditional methods of creating specialist LLMs require extensive fine-tuning and training of models on large datasets. Recently, prompt engineering, instead of fine-tuning, has shown potential to boost the performance of general foundation models. However, prompting methods such as chain-of-thought (CoT) may not be suitable for all subspecialty, and k-shot approaches may introduce irrelevant tokens into the context space. We present AutoMedPrompt, which explores the use of textual gradients to elicit medically relevant reasoning through system prompt optimization. AutoMedPrompt leverages TextGrad's automatic differentiation via text to improve the ability of general foundation LLMs. We evaluated AutoMedPrompt on Llama 3, an open-source LLM, using several QA benchmarks, including MedQA, PubMedQA, and the nephrology subspecialty-specific NephSAP. Our results show that prompting with textual gradients outperforms previous methods on open-source LLMs and surpasses proprietary models such as GPT-4, Claude 3 Opus, and Med-PaLM 2. AutoMedPrompt sets a new state-of-the-art (SOTA) performance on PubMedQA with an accuracy of 82.6%, while also outperforming previous prompting strategies on open-sourced models for MedQA (77.7%) and NephSAP (63.8%).