mlsanityData to 14 Oct 2026

Is ChatGPT a Biomedical Expert? -- Exploring the Zero-Shot Performance of Current GPT Models in Biomedical Tasks

arXiv 2306.16108 ↗First posted 28 Jun 2023cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

We assessed the performance of commercial Large Language Models (LLMs) GPT-3.5-Turbo and GPT-4 on tasks from the 2023 BioASQ challenge. In Task 11b Phase B, which is focused on answer generation, both models demonstrated competitive abilities with leading systems. Remarkably, they achieved this with simple zero-shot learning, grounded with relevant snippets. Even without relevant snippets, their performance was decent, though not on par with the best systems. Interestingly, the older and cheaper GPT-3.5-Turbo system was able to compete with GPT-4 in the grounded Q&A setting on factoid and list answers. In Task 11b Phase A, focusing on retrieval, query expansion through zero-shot learning improved performance, but the models fell short compared to other systems. The code needed to rerun these experiments is available through GitHub.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 17 of its 30 references that are in the map. First posted on arXiv 28 Jun 2023.

Builds on

17 of 30 references.

References · 17

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  5. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  6. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  7. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  8. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
  9. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  10. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,414
  11. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,045
  12. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 1,009
5 more
  1. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 449
  2. 2022Large Language Models Are Reasoning Teacherscited by 329
  3. 2021Search Strategy Formulation for Systematic Reviews: issues, challenges and opportunitiescited by 6
  4. 2021Interactive query expansion for professional search applicationscited by 5
  5. 2018Searching for Relevant Lessons Learned Using Hybrid Information Retrieval Classifiers: A Case Study in Software Engineeringcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 2; 2025: 1; 2026: 1.

5 papers.

  1. 2024Prompt engineering paradigms for medical applications: scoping review and recommendations for better practicescited by 14
  2. 2023On the Effectiveness of Creating Conversational Agent Personalities Through Promptingcited by 10
  3. 2024Can Open-Source LLMs Compete with Commercial Models? Exploring the Few-Shot Performance of Current GPT Models in Biomedical Taskscited by 6
  4. 2025Can Language Models Critique Themselves? Investigating Self-Feedback for Retrieval Augmented Generation at BioASQ 2025cited by 2
  5. 2026AnnoABSA: A Web-Based Annotation Tool for Aspect-Based Sentiment Analysis with Retrieval-Augmented Suggestionscited by 0