mlsanityData to 14 Oct 2026

Evaluation of ChatGPT on Biomedical Tasks: A Zero-Shot Comparison with Fine-Tuned Generative Transformers

arXiv 2306.04504 ↗First posted 7 Jun 2023cs.CL cs.LGcited by 22

Not on the map. Why?

Abstract (arXiv)

ChatGPT is a large language model developed by OpenAI. Despite its impressive performance across various tasks, no prior work has investigated its capability in the biomedical domain yet. To this end, this paper aims to evaluate the performance of ChatGPT on various benchmark biomedical tasks, such as relation extraction, document classification, question answering, and summarization. To the best of our knowledge, this is the first work that conducts an extensive evaluation of ChatGPT in the biomedical domain. Interestingly, we find based on our evaluation that in biomedical datasets that have smaller training sets, zero-shot ChatGPT even outperforms the state-of-the-art fine-tuned generative transformer models, such as BioGPT and BioBART. This suggests that ChatGPT's pre-training on large text corpora makes it quite specialized even in the biomedical domain. Our findings demonstrate that ChatGPT has the potential to be a valuable tool for various tasks in the biomedical domain that lack large annotated data.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Builds on 31 of its 45 references that are in the map. First posted on arXiv 7 Jun 2023.

Builds on

31 of 45 references.

References · 31

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  7. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  8. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
  9. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,414
  10. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  11. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 1,009
  12. 2019Publicly Available Clinical BERT Embeddingscited by 1,002
19 more
  1. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 894
  2. 2020A Primer in BERTology: What we know about how BERT workscited by 815
  3. 2022BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Miningcited by 538
  4. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 488
  5. 2023A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPTcited by 368
  6. 2019Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasetscited by 299
  7. 2020MedDialog: Two Large-scale Medical Dialogue Datasetscited by 161
  8. 2021A Unified Generative Framework for Various NER Subtaskscited by 141
  9. 2023A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasetscited by 123
  10. 2023Exploring the Limits of ChatGPT for Query or Aspect-based Text Summarizationcited by 122
  11. 2019A Survey on Recent Advances in Named Entity Recognition from Deep Learning modelscited by 117
  12. 2022BioBART: Pretraining and Evaluation of A Biomedical Generative Language Modelcited by 83
  13. 2020Question-Driven Summarization of Answers to Consumer Health Questionscited by 39
  14. 2022Generative Biomedical Entity Linking via Knowledge Base-Guided Pre-training and Synonyms-Aware Fine-tuningcited by 35
  15. 2021AMMU : A Survey of Transformer-based Biomedical Pretrained Language Modelscited by 32
  16. 2021Domain Adaptation with Pre-trained Transformers for Query Focused Abstractive Text Summarizationcited by 21
  17. 2021Discovering Drug-Target Interaction Knowledge from Biomedical Literaturecited by 14
  18. 2022BLINK with Elasticsearch for Efficient Entity Linking in Business Conversationscited by 13
  19. 2022Improving Named Entity Recognition in Telephone Conversations via Effective Active Learning with Human in the Loopcited by 7

Built on by

Cited by year

Citing papers per year: 2023: 4; 2024: 10; 2025: 8.

22 papers.

Sort:

By area:

  1. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  2. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 229
  3. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 163
  4. 2024A Survey on Retrieval-Augmented Text Generation for Large Language Modelscited by 77
  5. 2023A Comprehensive Evaluation of Large Language Models on Benchmark Biomedical Text Processing Taskscited by 36
  6. 2023BenLLMEval: A Comprehensive Evaluation into the Potentials and Pitfalls of Large Language Models on Bengali NLPcited by 23
  7. 2024Utilizing BERT for Information Retrieval: Survey, Applications, Resources, and Challengescited by 20
  8. 2025When Can We Trust LLMs in Mental Health? Large-Scale Benchmarks for Reliable LLM Evaluationcited by 19
  9. 2025Improving Automatic Evaluation of Large Language Models (LLMs) in Biomedical Relation Extraction via LLMs-as-the-Judgecited by 13
  10. 2025Performance Evaluation of Large Language Models in Bangla Consumer Health Query Summarizationcited by 6
  11. 2024An Evaluation of Large Language Models in Bioinformatics Researchcited by 5
  12. 2024Evaluating Search Engines and Large Language Models for Answering Health Questionscited by 5
10 more
  1. 2024Enhancing Text Annotation through Rationale-Driven Collaborative Few-Shot Promptingcited by 5
  2. 2024Comparative Analysis of Open-Source Language Models in Summarizing Medical Text Datacited by 4
  3. 2024LLM-IE: A Python Package for Generative Information Extraction with Large Language Modelscited by 4
  4. 2024Exploring ChatGPT for Next-generation Information Retrieval: Opportunities and Challengescited by 2
  5. 2025A Benchmark for End-to-End Zero-Shot Biomedical Relation Extraction with LLMs: Experiments with OpenAI Modelscited by 2
  6. 2025EvidenceBench: A Benchmark for Extracting Evidence from Biomedical Paperscited by 2
  7. 2025Exploring Zero-Shot App Review Classification with ChatGPT: Challenges and Potentialcited by 1
  8. 2024A Survey on the Real Power of ChatGPTcited by 0
  9. 2025Development of the user-friendly decision aid Rule-based Evaluation and Support Tool (REST) for optimizing the resources of an information extraction taskcited by 0
  10. 2025BIBERT-Pipe on Biomedical Nested Named Entity Linking at BioASQ 2025cited by 0