mlsanityData to 7 Oct 2026

Evaluation of ChatGPT Family of Models for Biomedical Reasoning and Classification

arXiv 2304.02496 ↗First posted 5 Apr 2023cs.CL cs.AIcited by 17

On the map: LLM reasoning → Clinical and biomedical applications

Abstract (arXiv)

Recent advances in large language models (LLMs) have shown impressive ability in biomedical question-answering, but have not been adequately investigated for more specific biomedical applications. This study investigates the performance of LLMs such as the ChatGPT family of models (GPT-3.5s, GPT-4) in biomedical tasks beyond question-answering. Because no patient data can be passed to the OpenAI API public interface, we evaluated model performance with over 10000 samples as proxies for two fundamental tasks in the clinical domain - classification and reasoning. The first task is classifying whether statements of clinical and policy recommendations in scientific literature constitute health advice. The second task is causal relation detection from the biomedical literature. We compared LLMs with simpler models, such as bag-of-words (BoW) with logistic regression, and fine-tuned BioBERT models. Despite the excitement around viral ChatGPT, we found that fine-tuning for two fundamental NLP tasks remained the best strategy. The simple BoW model performed on par with the most complex LLM prompting. Prompt engineering required significant investment.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Placed in Clinical and biomedical applications, one of 17 directions in LLM reasoning. Builds on 17 of its 34 references that are in the map. First posted on arXiv 5 Apr 2023.

Builds on

17 of 34 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Can large language models reason about medical questions?17 Jul

Other references · 16

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  7. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  8. 2022Emergent Abilities of Large Language Modelscited by 2,483
  9. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  10. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  11. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  12. 2022Galactica: A Large Language Model for Sciencecited by 736
4 more
  1. 2023On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspectivecited by 208
  2. 2023Do We Still Need Clinical Language Models?cited by 97
  3. 2023Translating Radiology Reports into Plain Language using ChatGPT and GPT-4 with Prompt Learning: Promising Results, Limitations, and Potentialcited by 39
  4. 2023Dr ChatGPT, tell me what I want to hear: How prompt knowledge impacts health answer correctnesscited by 15

Built on by

Cited by year

Citing papers per year: 2023: 4; 2024: 9; 2025: 4.

17 papers.

Sort:

  1. 2023Large Language Models to Identify Social Determinants of Health in Electronic Health Recordscited by 55
  2. 2024A Survey on Medical Large Language Models: Technology, Application, Trustworthiness, and Future Directionscited by 34
  3. 2023A Zero-shot and Few-shot Study of Instruction-Finetuned Large Language Models Applied to Clinical and Biomedical Taskscited by 25
  4. 2024MoralBERT: A Fine-Tuned Language Model for Capturing Moral Values in Social Discussionscited by 23
  5. 2023CohortGPT: An Enhanced GPT for Participant Recruitment in Clinical Studycited by 19
  6. 2024Assessing Generative Language Models in Classification Tasks: Performance and Self-Evaluation Capabilities in the Environmental and Climate Change Domaincited by 3
  7. 2024Large Language Models versus Classical Machine Learning: Performance in COVID-19 Mortality Prediction Using High-Dimensional Tabular Datacited by 3
  8. 2025Large Language Models for Healthcare Text Classification: A Systematic Reviewcited by 3
  9. 2024Improving Clinical NLP Performance through Language Model-Generated Synthetic Clinical Datacited by 2
  10. 2024Wait, but Tylenol is Acetaminophen... Investigating and Improving Language Models' Ability to Resist Requests for Misinformationcited by 2
  11. 2024Utilizing Large Language Models for Named Entity Recognition in Traditional Chinese Medicine against COVID-19 Literature: Comparative Studycited by 1
  12. 2025Automated Thematic Analyses Using LLMs: Xylazine Wound Management Social Media Chatter Use Casecited by 1
5 more
  1. 2023Measuring Pointwise V-Usable Information In-Context-lycited by 0
  2. 2024Coding historical causes of death data with Large Language Modelscited by 0
  3. 2024On Fusing ChatGPT and Ensemble Learning in Discon-tinuous Named Entity Recognition in Health Corporacited by 0
  4. 2025Review GIDE -- Restaurant Review Gastrointestinal Illness Detection and Extraction with Large Language Modelscited by 0
  5. 2025Cancer Diagnosis Categorization in Electronic Health Records Using Large Language Models and BioBERT: Model Performance Evaluation Studycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A Survey for Large Language Models in Biomedicine
  2. 2023Benchmarking large language models for biomedical natural language processing applications and recommendations
  3. 2023Does Synthetic Data Generation of LLMs Help Clinical Text Mining?
  4. 2023ClinicalGPT: Large Language Models Finetuned with Diverse Medical Data and Comprehensive Evaluation
  5. 2025Disentangling Reasoning and Knowledge in Medical Large Language Models
  6. 2024Can GPT Redefine Medical Understanding? Evaluating GPT on Biomedical Machine Reading Comprehension
  7. 2025Benchmarking GPT-5 for biomedical natural language processing
  8. 2023ChiMed-GPT: A Chinese Medical Large Language Model with Full Training Regime and Better Alignment to Human Preferences