mlsanityData to 7 Oct 2026

GatorTron: A Large Clinical Language Model to Unlock Patient Information from Unstructured Electronic Health Records

arXiv 2203.03540 ↗First posted 2 Feb 2022cs.CL cs.AI cs.LGcited by 238

Not on the map. Why?

Abstract (arXiv)

There is an increasing interest in developing artificial intelligence (AI) systems to process and interpret electronic health records (EHRs). Natural language processing (NLP) powered by pretrained language models is the key technology for medical AI systems utilizing clinical narratives. However, there are few clinical language models, the largest of which trained in the clinical domain is comparatively small at 110 million parameters (compared with billions of parameters in the general domain). It is not clear how large clinical language models with billions of parameters can help medical AI systems utilize unstructured EHRs. In this study, we develop from scratch a large clinical language model - GatorTron - using >90 billion words of text (including >82 billion words of de-identified clinical text) and systematically evaluate it on 5 clinical NLP tasks including clinical concept extraction, medical relation extraction, semantic textual similarity, natural language inference (NLI), and medical question answering (MQA). We examine how (1) scaling up the number of parameters and (2) scaling up the size of the training data could benefit these NLP tasks. GatorTron models scale up the clinical language model from 110 million to 8.9 billion parameters and improve 5 clinical NLP tasks (e.g., 9.6% and 9.5% improvement in accuracy for NLI and MQA), which can be applied to medical AI systems to improve healthcare delivery. The GatorTron models are publicly available at: https://catalog.ngc.nvidia.com/orgs/nvidia/teams/clara/models/gatortron_og.

Key facts. Cited by 238 papers in the map (data through 7 Oct 2026). Builds on 43 of its 86 references that are in the map. First posted on arXiv 2 Feb 2022.

Builds on

43 of 86 references.

References · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  6. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  7. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  8. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  9. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  10. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  11. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  12. 2015A large annotated corpus for learning natural language inferencecited by 1,837
31 more
  1. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  2. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  3. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  4. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  5. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 996
  6. 2019Publicly Available Clinical BERT Embeddingscited by 995
  7. 2020Efficient Transformers: A Surveycited by 784
  8. 2016Neural Architectures for Named Entity Recognitioncited by 548
  9. 2020Pre-trained Models for Natural Language Processing: A Surveycited by 504
  10. 2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 428
  11. 2020LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attentioncited by 326
  12. 2019SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimizationcited by 253
  13. 2019Dice Loss for Data-imbalanced NLP Taskscited by 180
  14. 2020Named Entity Recognition as Dependency Parsingcited by 150
  15. 2018emrQA: A Large Corpus for Question Answering on Electronic Medical Recordscited by 134
  16. 2019A Question-Entailment Approach to Question Answeringcited by 133
  17. 2021Entailment as Few-Shot Learnercited by 122
  18. 2019Semantics-aware BERT for Language Understandingcited by 111
  19. 2020Two are Better than One: Joint Entity and Relation Extraction with Table-Sequence Encoderscited by 87
  20. 2020Retrospective Reader for Machine Reading Comprehensioncited by 86
  21. 2020BioMegatron: Larger Biomedical Domain Language Modelcited by 76
  22. 2019TANDA: Transfer and Adapt Pre-Trained Transformer Models for Answer Sentence Selectioncited by 74
  23. 2021Entity Structure Within and Throughout: Modeling Mention Dependencies for Document-Level Relation Extractioncited by 68
  24. 2018MedSTS: A Resource for Clinical Semantic Textual Similaritycited by 39
  25. 2020The Depth-to-Width Interplay in Self-Attentioncited by 39
  26. 2021Relation Classification with Entity Type Restrictioncited by 31
  27. 2020Clinical Reading Comprehension: A Thorough Analysis of the emrQA Datasetcited by 23
  28. 2017A Survey of Deep Learning Methods for Relation Extractioncited by 16
  29. 2019Measuring Sentences Similarity: A Surveycited by 9
  30. 2021Clinical Relation Extraction Using Transformer-based Modelscited by 9
  31. 2021Extracting Adverse Drug Events from Clinical Notescited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 59; 2024: 74; 2025: 57; 2026: 47.

238 papers.

Sort:

By area:

  1. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  2. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  3. 2024Foundation Models for Time Series Analysis: A Tutorial and Surveycited by 234
  4. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 162
  5. 2023A Study of Generative Large Language Model for Medical Research and Healthcarecited by 160
  6. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 134
  7. 2023Large language models in healthcare and medical domain: A reviewcited by 126
  8. 2023A Survey of Large Language Models in Medicine: Progress, Application, and Challengecited by 123
  9. 2024Vision-Language Models for Medical Report Generation and Visual Question Answering: A Reviewcited by 121
  10. 2023Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlookcited by 109
  11. 2023Do We Still Need Clinical Language Models?cited by 97
  12. 2024Foundation Model for Advancing Healthcare: Challenges, Opportunities, and Future Directionscited by 77
38 more
  1. 2024EHRAgent: Code Empowers Large Language Models for Few-shot Complex Tabular Reasoning on Electronic Health Recordscited by 75
  2. 2024KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantizationcited by 74
  3. 2023Large Language Models and Knowledge Graphs: Opportunities and Challengescited by 71
  4. 2024A Complete Survey on LLM-based AI Chatbotscited by 71
  5. 2023Large AI Models in Health Informatics: Applications, Challenges, and the Futurecited by 61
  6. 2024Empowering Time Series Analysis with Large Language Models: A Surveycited by 60
  7. 2024A Comprehensive Survey of Foundation Models in Medicinecited by 56
  8. 2023Publicly Shareable Clinical Large Language Model Built on Synthetic Clinical Notescited by 51
  9. 2023Multimodal LLMs for health grounded in individual-specific datacited by 48
  10. 2024RareBench: Can LLMs Serve as Rare Diseases Specialists?cited by 45
  11. 2024A Toolbox for Surfacing Health Equity Harms and Biases in Large Language Modelscited by 45
  12. 2024A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicinecited by 45
  13. 2023Training Data Extraction From Pre-trained Language Models: A Surveycited by 43
  14. 2023Benefits and Harms of Large Language Models in Digital Mental Healthcited by 39
  15. 2023Parameter-Efficient Fine-Tuning of LLaMA for the Clinical Domaincited by 38
  16. 2024Apollo: A Lightweight Multilingual Medical LLM towards Democratizing Medical AI to 6B Peoplecited by 38
  17. 2024Large Language Models for Medicine: A Surveycited by 38
  18. 2023Taiyi: A Bilingual Fine-Tuned Large Language Model for Diverse Biomedical Taskscited by 36
  19. 2023CPLLM: Clinical Prediction with Large Language Modelscited by 34
  20. 2024A Survey on Medical Large Language Models: Technology, Application, Trustworthiness, and Future Directionscited by 34
  21. 2024EHRNoteQA: An LLM Benchmark for Real-World Clinical Practice Using Discharge Summariescited by 33
  22. 2023Model Tuning or Prompt Tuning? A Study of Large Language Models for Clinical Concept and Relation Extractioncited by 32
  23. 2024HealthQ: Unveiling Questioning Capabilities of LLM Chains in Healthcare Conversationscited by 31
  24. 2023Identifying and Mitigating Privacy Risks Stemming from Language Models: A Surveycited by 30
  25. 2023Computational Pathology at Health System Scale -- Self-Supervised Foundation Models from Three Billion Imagescited by 30
  26. 2023DrBERT: A Robust Pre-trained Model in French for Biomedical and Clinical domainscited by 27
  27. 2024Weaver: Foundation Models for Creative Writingcited by 25
  28. 2023Enhancing Psychological Counseling with Large Language Model: A Multifaceted Decision-Support System for Non-Professionalscited by 24
  29. 2023Certified Robustness for Large Language Models with Self-Denoisingcited by 23
  30. 2023Large language models in medicine: the potentials and pitfallscited by 21
  31. 2024Leveraging Large Language Model as Simulated Patients for Clinical Educationcited by 20
  32. 2024OLAPH: Improving Factuality in Biomedical Long-form Question Answeringcited by 20
  33. 2025Foundation Models for Geospatial Reasoning: Assessing Capabilities of Large Language Models in Understanding Geometries and Topological Spatial Relationscited by 20
  34. 2023Transformers in Healthcare: A Surveycited by 17
  35. 2023Enhancing Phenotype Recognition in Clinical Notes Using Large Language Models: PhenoBCBERT and PhenoGPTcited by 17
  36. 2024Chain-of-Scrutiny: Detecting Backdoor Attacks for Large Language Modelscited by 17
  37. 2023An Interdisciplinary Outlook on Large Language Models for Scientific Researchcited by 15
  38. 2023Improving accuracy of GPT-3/4 results on biomedical data using a retrieval-augmented language modelcited by 14

50 most cited of 238 All 238 (JSON)