mlsanityData to 7 Oct 2026

SciFive: a text-to-text transformer model for biomedical literature

arXiv 2106.03598 ↗First posted 28 May 2021cs.CL cs.AI cs.LGcited by 99

Not on the map. Why?

Abstract (arXiv)

In this report, we introduce SciFive, a domain-specific T5 model that has been pre-trained on large biomedical corpora. Our model outperforms the current SOTA methods (i.e. BERT, BioBERT, Base T5) on tasks in named entity relation, relation extraction, natural language inference, and question-answering. We show that text-generation methods have significant potential in a broad array of biomedical NLP tasks, particularly those requiring longer, more complex outputs. Our results support the exploration of more difficult text generation tasks and the development of new methods in this area

Key facts. Cited by 99 papers in the map (data through 7 Oct 2026). Builds on 9 of its 23 references that are in the map. First posted on arXiv 28 May 2021.

Builds on

9 of 23 references.

References · 9

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  5. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  6. 2017An Overview of Multi-Task Learning in Deep Neural Networkscited by 791
  7. 2019Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasetscited by 298
  8. 2018Lessons from Natural Language Inference in the Clinical Domaincited by 149
  9. 2018Clinical Concept Extraction with Contextual Word Embeddingcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 22; 2023: 30; 2024: 24; 2025: 14; 2026: 6.

99 papers.

Sort:

By area:

  1. 2022BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Miningcited by 531
  2. 2023BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Taskscited by 201
  3. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 162
  4. 2023A Survey of Large Language Models in Medicine: Progress, Application, and Challengecited by 123
  5. 2023Do We Still Need Clinical Language Models?cited by 97
  6. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
  7. 2021Pre-trained Language Models in Biomedical Domain: A Systematic Surveycited by 87
  8. 2022Thinking about GPT-3 In-Context Learning for Biomedical IE? Think Againcited by 83
  9. 2022BioBART: Pretraining and Evaluation of A Biomedical Generative Language Modelcited by 83
  10. 2023Towards Mitigating Hallucination in Large Language Models via Self-Reflectioncited by 81
  11. 2023Benchmarking large language models for biomedical natural language processing applications and recommendationscited by 78
  12. 2023Matching Patients to Clinical Trials with Large Language Modelscited by 68
38 more
  1. 2024MedExpQA: Multilingual Benchmarking of Large Language Models for Medical Question Answeringcited by 43
  2. 2022ViT5: Pretrained Text-to-Text Transformer for Vietnamese Language Generationcited by 37
  3. 2022BigBIO: A Framework for Data-Centric Biomedical Natural Language Processingcited by 36
  4. 2022PHEE: A Dataset for Pharmacovigilance Event Extraction from Textcited by 36
  5. 2023MatSci-NLP: Evaluating Scientific Language Models on Materials Science Language Tasks Using Text-to-Schema Modelingcited by 35
  6. 2022Learning to Revise References for Faithful Summarizationcited by 28
  7. 2023nach0: Multimodal Natural and Chemical Languages Foundation Modelcited by 26
  8. 2022A sequence-to-sequence approach for document-level relation extractioncited by 25
  9. 2023RadAdapt: Radiology Report Summarization via Lightweight Domain Adaptation of Large Language Modelscited by 24
  10. 2022SpaBERT: A Pretrained Language Model from Geographic Data for Geo-Entity Representationcited by 23
  11. 2025Integrating LLMs with ITS: Recent Advances, Potentials, Challenges, and Future Directionscited by 23
  12. 2021Rewire-then-Probe: A Contrastive Recipe for Probing Biomedical Knowledge of Pre-trained Language Modelscited by 21
  13. 2024Unveiling LLM Evaluation Focused on Metrics: Challenges and Solutionscited by 17
  14. 2022In-BoXBART: Get Instructions into Biomedical Multi-Task Learningcited by 16
  15. 2023Med-EASi: Finely Annotated Dataset and Models for Controllable Simplification of Medical Textscited by 16
  16. 2024Leveraging Biomolecule and Natural Language through Multi-Modal Learning: A Surveycited by 16
  17. 2024Medical mT5: An Open-Source Multilingual Text-to-Text LLM for The Medical Domaincited by 15
  18. 2022DR.BENCH: Diagnostic Reasoning Benchmark for Clinical Natural Language Processingcited by 14
  19. 2022Can NLI Provide Proper Indirect Supervision for Low-resource Biomedical Relation Extraction?cited by 13
  20. 2024Open Challenges and Opportunities in Federated Foundation Models Towards Biomedical Healthcarecited by 12
  21. 2025MedCite: Can Language Models Generate Verifiable Text for Medicine?cited by 11
  22. 2023NERetrieve: Dataset for Next Generation Named Entity Recognition and Retrievalcited by 10
  23. 2024GP-GPT: Large Language Model for Gene-Phenotype Mappingcited by 10
  24. 2022Generalizing over Long Tail Concepts for Medical Term Normalizationcited by 8
  25. 2023What are the Desired Characteristics of Calibration Sets? Identifying Correlates on Long Form Scientific Summarizationcited by 8
  26. 2023Prompt-based Extraction of Social Determinants of Health Using Few-shot Learningcited by 8
  27. 2023Investigating Large Language Models and Control Mechanisms to Improve Text Readability of Biomedical Abstractscited by 8
  28. 2024BioMamba: Domain-Adaptive Biomedical Language Modelscited by 8
  29. 2021Deeper Clinical Document Understanding Using Relation Extractioncited by 7
  30. 2024A Survey of Pre-trained Language Models for Processing Scientific Textcited by 7
  31. 2024Exploration of Masked and Causal Language Modelling for Text Generationcited by 7
  32. 2024MedINST: Meta Dataset of Biomedical Instructionscited by 7
  33. 2022Comparison of biomedical relationship extraction methods and models for knowledge graph creationcited by 6
  34. 2023Compositional Zero-Shot Domain Transfer with Text-to-Text Modelscited by 6
  35. 2023SemEval-2023 Task 7: Multi-Evidence Natural Language Inference for Clinical Trial Datacited by 6
  36. 2023Explanatory Argument Extraction of Correct Answers in Resident Medical Examscited by 6
  37. 2024Clinical Insights: A Comprehensive Review of Language Models in Medicinecited by 6
  38. 2025Scaling Up Biomedical Vision-Language Models: Fine-Tuning, Instruction Tuning, and Multi-Modal Learningcited by 6

50 most cited of 99 All 99 (JSON)