mlsanityData to 14 Oct 2026

Employing General-Purpose and Biomedical Large Language Models with Advanced Prompt Engineering for Pharmacoepidemiologic Study Design

arXiv 2604.17988 ↗First posted 20 Apr 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Background: The potential of large language models (LLMs) to automate and support pharmacoepidemiologic study design is an emerging area of interest, yet their reliability remains insufficiently characterized. General-purpose LLMs often display inaccuracies, while the comparative performance of specialized biomedical LLMs in this domain remains unknown. Methods: This study evaluated general-purpose LLMs (GPT-4o and DeepSeek-R1) versus biomedically fine-tuned LLMs (QuantFactory/Bio-Medical-Llama-3-8B-GGUF and Irathernotsay/qwen2-1.5B-medical_qa-Finetune) using 46 protocols (2018-2024) from the HMA-EMA Catalogue and Sentinel System. Performance was assessed across relevance, logic of justification, and ontology-code agreement across multiple coding systems using Least-to-Most (LTM) and Active Prompting strategies. Results: GPT-4o and DeepSeek-R1 paired with LTM prompting achieved the highest relevance and logic of justification scores, with GPT-4o-LTM reaching a median relevance score of 4 in 8 of 9 questions for HMA-EMA protocols. Biomedical LLMs showed lower relevance overall and frequently generated insufficient justification. All LLMs demonstrated limited proficiency in ontology-code mapping, although LTM provided the most consistent improvements in reasoning stability. Conclusion: Off-the-shelf general-purpose LLMs currently offer superior support for pharmacoepidemiologic design compared to biomedical LLMs. Prompt strategy strongly influenced LLM performance.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 5 of its 26 references that are in the map. First posted on arXiv 20 Apr 2026.

Builds on

5 of 26 references.

References · 5

  1. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,407
  2. 2024A Survey of Prompt Engineering Methods in Large Language Models for Different NLP Taskscited by 47
  3. 2024Does Biomedical Training Lead to Better Medical Performance?cited by 13
  4. 2024Can GPT Improve the State of Prior Authorization via Guideline Based Automated Question Answering?cited by 7
  5. 2018Epidemiological data challenges: planning for a more robust future through data standardscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Biomedical Large Languages Models Seem not to be Superior to Generalist Models on Unseen Medical Data
  2. 2024Entry-level guide to the use of large language models for medical research
  3. 2024A Survey for Large Language Models in Biomedicine
  4. 2024Can Large Language Models Replace Data Scientists in Biomedical Research?
  5. 2026A primer on evaluation methods for large language models in healthcare
  6. 2024A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicine
  7. 2025A Scoping Review of Synthetic Data Generation by Language Models in Biomedical Research and Application: Data Utility and Quality Perspectives
  8. 2025AutoMedPrompt: A New Framework for Optimizing LLM Medical Prompts Using Textual Gradients