mlsanityData to 7 Oct 2026

BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining

arXiv 2210.10341 ↗First posted 19 Oct 2022cs.CL cs.AIcited by 531

Not on the map. Why?

Cited by 32 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Pre-trained language models have attracted increasing attention in the biomedical domain, inspired by their great success in the general natural language domain. Among the two main branches of pre-trained language models in the general language domain, i.e., BERT (and its variants) and GPT (and its variants), the first one has been extensively studied in the biomedical domain, such as BioBERT and PubMedBERT. While they have achieved great success on a variety of discriminative downstream biomedical tasks, the lack of generation ability constrains their application scope. In this paper, we propose BioGPT, a domain-specific generative Transformer language model pre-trained on large scale biomedical literature. We evaluate BioGPT on six biomedical NLP tasks and demonstrate that our model outperforms previous models on most tasks. Especially, we get 44.98%, 38.42% and 40.76% F1 score on BC5CDR, KD-DTI and DDI end-to-end relation extraction tasks respectively, and 78.2% accuracy on PubMedQA, creating a new record. Our case study on text generation further demonstrates the advantage of BioGPT on biomedical literature to generate fluent descriptions for biomedical terms. Code is available at https://github.com/microsoft/BioGPT.

Key facts. Cited by 531 papers in the map (data through 7 Oct 2026). Builds on 33 of its 59 references that are in the map. First posted on arXiv 19 Oct 2022.

Builds on

33 of 59 references.

References · 33

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  9. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  10. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  11. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  12. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
21 more
  1. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  2. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  3. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 996
  4. 2020SPECTER: Document-level Representation Learning using Citation-informed Transformerscited by 426
  5. 2020LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attentioncited by 326
  6. 2019Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasetscited by 298
  7. 2022LinkBERT: Pretraining Language Models with Document Linkscited by 209
  8. 2018QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehensioncited by 164
  9. 2021SciFive: a text-to-text transformer model for biomedical literaturecited by 99
  10. 2022Thinking about GPT-3 In-Context Learning for Biomedical IE? Think Againcited by 83
  11. 2020TPLinker: Single-stage Joint Extraction of Entities and Relations Through Token Pair Linkingcited by 72
  12. 2020Joint Entity and Relation Extraction with Set Prediction Networkscited by 54
  13. 2020DARE: Data Augmented Relation Extraction with GPT-2cited by 43
  14. 2020Global-to-Local Neural Networks for Document-Level Relation Extractioncited by 43
  15. 2021A Partition Filter Network for Joint Entity and Relation Extractioncited by 42
  16. 2021GPT-3 Models are Poor Few-Shot Learners in the Biomedical Domaincited by 36
  17. 2020Results of the seventh edition of the BioASQ Challengecited by 28
  18. 2020Minimize Exposure Bias of Seq2Seq Models in Joint Entity and Relation Extractioncited by 27
  19. 2022A sequence-to-sequence approach for document-level relation extractioncited by 25
  20. 2021Discovering Drug-Target Interaction Knowledge from Biomedical Literaturecited by 14
  21. 2021ELECTRAMed: a new pre-trained language representation model for biomedical NLPcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 135; 2024: 140; 2025: 159; 2026: 94.

531 papers.

Sort:

By area:

  1. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  2. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  3. 2023BloombergGPT: A Large Language Model for Financecited by 923
  4. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 738
  5. 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 506
  6. 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 387
  7. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 385
  8. 2023Towards Expert-Level Medical Question Answering with Large Language Modelscited by 382
  9. 2024Benchmarking Retrieval-Augmented Generation for Medicinecited by 326
  10. 2023Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicinecited by 319
  11. 2024BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domainscited by 299
  12. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
38 more
  1. 2023HuatuoGPT, towards Taming Language Model to Be a Doctorcited by 211
  2. 2022Is GPT-3 a Good Data Annotator?cited by 208
  3. 2023On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspectivecited by 208
  4. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 204
  5. 2023BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Taskscited by 201
  6. 2024SciCode: A Research Coding Benchmark Curated by Scientistscited by 172
  7. 2023A Study of Generative Large Language Model for Medical Research and Healthcarecited by 160
  8. 2024GraphRouter: A Graph-based Router for LLM Selectionscited by 154
  9. 2023PaperQA: Retrieval-Augmented Generative Agent for Scientific Researchcited by 143
  10. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 134
  11. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 129
  12. 2023DeID-GPT: Zero-shot Medical Text De-Identification by GPT-4cited by 128
  13. 2023Emergent autonomous scientific research capabilities of large language modelscited by 127
  14. 2023Large language models in healthcare and medical domain: A reviewcited by 126
  15. 2023A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasetscited by 123
  16. 2023A Survey of Large Language Models in Medicine: Progress, Application, and Challengecited by 123
  17. 2023Almanac: Retrieval-Augmented Language Models for Clinical Medicinecited by 122
  18. 2023Opportunities and Challenges for ChatGPT and Large Language Models in Biomedicine and Healthcited by 118
  19. 2024Empowering Biomedical Discovery with AI Agentscited by 117
  20. 2023GeneGPT: Augmenting Large Language Models with Domain Tools for Improved Access to Biomedical Informationcited by 115
  21. 2023SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific Researchcited by 112
  22. 2023BioT5: Enriching Cross-modal Integration in Biology with Chemical Knowledge and Natural Language Associationscited by 100
  23. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
  24. 2025Towards Scientific Intelligence: A Survey of LLM-based Scientific Agentscited by 96
  25. 2023Conversational Health Agents: A Personalized LLM-Powered Agent Frameworkcited by 91
  26. 2024Medical Graph RAG: Towards Safe Medical Large Language Model via Graph Retrieval-Augmented Generationcited by 89
  27. 2023PMC-LLaMA: Towards Building Open-source Language Models for Medicinecited by 87
  28. 2024BioMedLM: A 2.7B Parameter Language Model Trained On Biomedical Textcited by 85
  29. 2023GIT-Mol: A Multi-modal Large Language Model for Molecular Science with Graph, Image, and Textcited by 82
  30. 2023MolXPT: Wrapping Molecules with Text for Generative Pre-trainingcited by 81
  31. 2024PRISM: A Multi-Modal Generative Foundation Model for Slide-Level Histopathologycited by 80
  32. 2023Benchmarking large language models for biomedical natural language processing applications and recommendationscited by 78
  33. 2023K2: A Foundation Language Model for Geoscience Knowledge Understanding and Utilizationcited by 69
  34. 2024FairCLIP: Harnessing Fairness in Vision-Language Learningcited by 69
  35. 2023FinGPT: Democratizing Internet-scale Data for Financial Large Language Modelscited by 65
  36. 2023Knowledge-Augmented Reasoning Distillation for Small Language Models in Knowledge-Intensive Taskscited by 62
  37. 2023Large AI Models in Health Informatics: Applications, Challenges, and the Futurecited by 61
  38. 2023Are ChatGPT and GPT-4 General-Purpose Solvers for Financial Text Analytics? A Study on Several Typical Taskscited by 59

50 most cited of 531 All 531 (JSON)