# Employing General-Purpose and Biomedical Large Language Models with Advanced Prompt Engineering for Pharmacoepidemiologic Study Design

arXiv: https://arxiv.org/abs/2604.17988 · HTML: https://mlsanity.com/paper/2604.17988/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 5 of its 26 references that are in the map.
- First posted on arXiv 20 Apr 2026.

## Builds on (most cited first, 5 of 5)

- [Least-to-Most Prompting Enables Complex Reasoning in Large Language Models](https://mlsanity.com/paper/2205.10625/) (2022)
- [A Survey of Prompt Engineering Methods in Large Language Models for Different NLP Tasks](https://mlsanity.com/paper/2407.12994/) (2024)
- [Does Biomedical Training Lead to Better Medical Performance?](https://mlsanity.com/paper/2404.04067/) (2024)
- [Can GPT Improve the State of Prior Authorization via Guideline Based Automated Question Answering?](https://mlsanity.com/paper/2402.18419/) (2024)
- [Epidemiological data challenges: planning for a more robust future through data standards](https://mlsanity.com/paper/1805.00445/) (2018)

## Abstract (arXiv)

Background: The potential of large language models (LLMs) to automate and support pharmacoepidemiologic study design is an emerging area of interest, yet their reliability remains insufficiently characterized. General-purpose LLMs often display inaccuracies, while the comparative performance of specialized biomedical LLMs in this domain remains unknown. Methods: This study evaluated general-purpose LLMs (GPT-4o and DeepSeek-R1) versus biomedically fine-tuned LLMs (QuantFactory/Bio-Medical-Llama-3-8B-GGUF and Irathernotsay/qwen2-1.5B-medical_qa-Finetune) using 46 protocols (2018-2024) from the HMA-EMA Catalogue and Sentinel System. Performance was assessed across relevance, logic of justification, and ontology-code agreement across multiple coding systems using Least-to-Most (LTM) and Active Prompting strategies. Results: GPT-4o and DeepSeek-R1 paired with LTM prompting achieved the highest relevance and logic of justification scores, with GPT-4o-LTM reaching a median relevance score of 4 in 8 of 9 questions for HMA-EMA protocols. Biomedical LLMs showed lower relevance overall and frequently generated insufficient justification. All LLMs demonstrated limited proficiency in ontology-code mapping, although LTM provided the most consistent improvements in reasoning stability. Conclusion: Off-the-shelf general-purpose LLMs currently offer superior support for pharmacoepidemiologic design compared to biomedical LLMs. Prompt strategy strongly influenced LLM performance.