mlsanityData to 13 Oct 2026

A Zero-shot and Few-shot Study of Instruction-Finetuned Large Language Models Applied to Clinical and Biomedical Tasks

arXiv 2307.12114 ↗First posted 22 Jul 2023cs.CL cs.AI cs.LGcited by 25

Not on the map. Why?

Abstract (arXiv)

We evaluate four state-of-the-art instruction-tuned large language models (LLMs) -- ChatGPT, Flan-T5 UL2, Tk-Instruct, and Alpaca -- on a set of 13 real-world clinical and biomedical natural language processing (NLP) tasks in English, such as named-entity recognition (NER), question-answering (QA), relation extraction (RE), etc. Our overall results demonstrate that the evaluated LLMs begin to approach performance of state-of-the-art models in zero- and few-shot scenarios for most tasks, and particularly well for the QA task, even though they have never seen examples from these tasks before. However, we observed that the classification and RE tasks perform below what can be achieved with a specifically trained model for the medical field, such as PubMedBERT. Finally, we noted that no LLM outperforms all the others on all the studied tasks, with some models being better suited for certain tasks than others.

Key facts. Cited by 25 papers in the map (data through 13 Oct 2026). Builds on 34 of its 59 references that are in the map. First posted on arXiv 22 Jul 2023.

Builds on

34 of 59 references.

References · 34

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  2. 2022Training language models to follow instructions with human feedbackcited by 15,856
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,549
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,072
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,107
  6. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,480
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,317
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,765
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,500
  10. 2022Scaling Instruction-Finetuned Language Modelscited by 3,007
  11. 2022Training Compute-Optimal Large Language Modelscited by 2,972
  12. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,379
22 more
  1. 2022Large Language Models Encode Clinical Knowledgecited by 1,817
  2. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,468
  3. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,338
  4. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 1,001
  5. 2019How Can We Know What Language Models Know?cited by 958
  6. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 935
  7. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 894
  8. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 865
  9. 2017Crowdsourcing Multiple Choice Science Questionscited by 751
  10. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  11. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 697
  12. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  13. 2023A Comprehensive Capability Analysis of GPT-3 and GPT-3.5 Series Modelscited by 246
  14. 2020Explainable Automated Fact-Checking for Public Health Claimscited by 211
  15. 2021Reframing Instructional Prompts to GPTk's Languagecited by 176
  16. 2022Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanationscited by 159
  17. 2020The Turking Test: Can Language Models Understand Instructions?cited by 82
  18. 2023Evaluation of ChatGPT for NLP-based Mental Health Applicationscited by 53
  19. 2020Evidence Inference 2.0: More Data, Better Modelscited by 40
  20. 2023Evaluation of ChatGPT Family of Models for Biomedical Reasoning and Classificationcited by 17
  21. 2023Evaluation of GPT and BERT-based models on identifying protein-protein interactions in biomedical textcited by 9
  22. 2023Data Augmentation for Robust Character Detection in Fantasy Novelscited by 1

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2023: 3; 2024: 6; 2025: 9; 2026: 7.

25 papers.

Sort:

By area:

  1. 2023Large Language Models for Generative Information Extraction: A Surveycited by 184
  2. 2023Evaluating Instruction-Tuned Large Language Models on Code Comprehension and Generationcited by 50
  3. 2024Language Models for Text Classification: Is In-Context Learning Enough?cited by 39
  4. 2024ULTRA: Unleash LLMs' Potential for Event Argument Extraction through Hierarchical Modeling and Pair-wise Self-Refinementcited by 12
  5. 2025Problem Solved? Information Extraction Design Space for Layout-Rich Documents using LLMscited by 6
  6. 2024ProtoMed-LLM: An Automatic Evaluation Framework for Large Language Models in Medical Protocol Formulationcited by 4
  7. 2024Adaptation of Biomedical and Clinical Pretrained Models to French Long Documents: A Comparative Studycited by 2
  8. 2025Retrieval augmented generation based dynamic prompting for few-shot biomedical named entity recognition using large language modelscited by 2
  9. 2023An Eye on Clinical BERT: Investigating Language Model Generalization for Diabetic Eye Disease Phenotypingcited by 1
  10. 2025Are LLMs Enough for Hyperpartisan, Fake, Polarized and Harmful Content Detection? Evaluating In-Context Learning vs. Fine-Tuningcited by 1
  11. 2026Few-Shot Large Language Models for Actionable Triage Categorization of Online Patient Inquiriescited by 1
  12. 2024Prompt2Fashion: An automatically generated fashion datasetcited by 0
13 more
  1. 2024From Words to Workflows: Automating Business Processescited by 0
  2. 2025Synthesized Annotation Guidelines are Knowledge-Lite Boosters for Clinical Information Extractioncited by 0
  3. 2025Entity-Augmented Neuroscience Knowledge Retrieval Using Ontology and Semantic Understanding Capability of LLMcited by 0
  4. 2025A Large Language Model Based Pipeline for Review of Systems Entity Recognition from Clinical Notescited by 0
  5. 2025Development of the user-friendly decision aid Rule-based Evaluation and Support Tool (REST) for optimizing the resources of an information extraction taskcited by 0
  6. 2025Small Language Models for Emergency Departments Decision Support: A Benchmark Studycited by 0
  7. 2025CoLM: Collaborative Large Models via A Client-Server Paradigmcited by 0
  8. 2026Personalized Prediction of Perceived Message Effectiveness Using Large Language Model Based Digital Twinscited by 0
  9. 2026AnnoABSA: A Web-Based Annotation Tool for Aspect-Based Sentiment Analysis with Retrieval-Augmented Suggestionscited by 0
  10. 2026Automatic Cardiac Risk Management Classification using large-context Electronic Patients Health Recordscited by 0
  11. 2026The GELATO Dataset for Legislative NERcited by 0
  12. 2026Automated Detection and Classification of Delusion-related Content in Naturalistic Audio Diaries Using Multi-Agent Language Modelscited by 0
  13. 2026Large Language Models for Automated Cross-Domain Machine Learning Task Type Identification: A Benchmark Dataset and Evaluationcited by 0