# Enhancing Phenotype Recognition in Clinical Notes Using Large Language Models: PhenoBCBERT and PhenoGPT

arXiv: https://arxiv.org/abs/2308.06294 · HTML: https://mlsanity.com/paper/2308.06294/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 72 references that are in the map.
- First posted on arXiv 11 Aug 2023.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale](https://mlsanity.com/paper/2208.07339/) (2022)
- [The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only](https://mlsanity.com/paper/2306.01116/) (2023)
- [Large Language Models Can Self-Improve](https://mlsanity.com/paper/2210.11610/) (2022)
- [Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease prediction](https://mlsanity.com/paper/2005.12833/) (2020)
- [GatorTron: A Large Clinical Language Model to Unlock Patient Information from Unstructured Electronic Health Records](https://mlsanity.com/paper/2203.03540/) (2022)
- [CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation](https://mlsanity.com/paper/2103.06874/) (2021)
- [Predictability and Surprise in Large Generative Models](https://mlsanity.com/paper/2202.07785/) (2022)
- [Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing](https://mlsanity.com/paper/2006.03236/) (2020)
- [Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?](https://mlsanity.com/paper/2104.07762/) (2021)
- [Named Entity Recognition without Labelled Data: A Weak Supervision Approach](https://mlsanity.com/paper/2004.14723/) (2020)
- [BERN2: an advanced neural biomedical named entity recognition and normalization tool](https://mlsanity.com/paper/2201.02080/) (2022)
- [Named Entity Recognition with Small Strongly Labeled and Large Weakly Labeled Data](https://mlsanity.com/paper/2106.08977/) (2021)
- [Better Word Embeddings by Disentangling Contextual n-Gram Information](https://mlsanity.com/paper/1904.05033/) (2019)
- [PhenoTagger: A Hybrid Method for Phenotype Concept Recognition using Human Phenotype Ontology](https://mlsanity.com/paper/2009.08478/) (2020)
- [Misspelling Correction with Pre-trained Contextual Language Model](https://mlsanity.com/paper/2101.03204/) (2021)
- [NCBO Ontology Recommender 2.0: An Enhanced Approach for Biomedical Ontology Recommendation](https://mlsanity.com/paper/1611.05973/) (2016)

## Built on by (most cited first, 18 of 18)

- [RareBench: Can LLMs Serve as Rare Diseases Specialists?](https://mlsanity.com/paper/2402.06341/) (2024)
- [Distilling Large Language Models for Matching Patients to Clinical Trials](https://mlsanity.com/paper/2312.09958/) (2023)
- [Few-shot clinical entity recognition in English, French and Spanish: masked language models outperform generative model prompting](https://mlsanity.com/paper/2402.12801/) (2024)
- [AutoPCR: Automated Phenotype Concept Recognition by Prompting](https://mlsanity.com/paper/2507.19315/) (2025)
- [Large Language Models Struggle in Token-Level Clinical Named Entity Recognition](https://mlsanity.com/paper/2407.00731/) (2024)
- [FactEHR: A Dataset for Evaluating Factuality in Clinical Notes Using LLMs](https://mlsanity.com/paper/2412.12422/) (2024)
- [GestaltMML: Enhancing Rare Genetic Disease Diagnosis through Multimodal Machine Learning Combining Facial Images and Clinical Text](https://mlsanity.com/paper/2312.15320/) (2023)
- [A Large Language Model Outperforms Other Computational Approaches to the High-Throughput Phenotyping of Physician Notes](https://mlsanity.com/paper/2406.14757/) (2024)
- [Can GPT Redefine Medical Understanding? Evaluating GPT on Biomedical Machine Reading Comprehension](https://mlsanity.com/paper/2405.18682/) (2024)
- [From Fuzzy Speech to Medical Insight: Benchmarking LLMs on Noisy Patient Narratives](https://mlsanity.com/paper/2509.11803/) (2025)
- [Leveraging Large Language Models and Survival Analysis for Early Prediction of Chemotherapy Outcomes](https://mlsanity.com/paper/2603.11594/) (2026)
- [A survey of using EHR as real-world evidence for discovering and validating new drug indications](https://mlsanity.com/paper/2505.24767/) (2025)
- [A Generative Approach for Semantic Auditing of Electronic Health Records](https://mlsanity.com/paper/2507.02628/) (2025)
- [RDMA: Cost Effective Agent-Driven Rare Disease Mining from Electronic Health Records](https://mlsanity.com/paper/2507.15867/) (2025)
- [An artificial intelligence framework for end-to-end rare disease phenotyping from clinical notes using large language models](https://mlsanity.com/paper/2602.20324/) (2026)
- [A specialized reasoning large language model for accelerating rare disease diagnosis: a randomized AI physician assistance trial](https://mlsanity.com/paper/2606.24510/) (2026)
- [Toward Automated Detection of Documentation Inconsistencies in Electronic Health Records](https://mlsanity.com/paper/2607.22954/) (2026)
- [UMCR: Universal Medical Concept Recognition through Guideline Integration and Agentic Linking](https://mlsanity.com/paper/2610.17363/) (2026)

## Abstract (arXiv)

We hypothesize that large language models (LLMs) based on the transformer architecture can enable automated detection of clinical phenotype terms, including terms not documented in the HPO. In this study, we developed two types of models: PhenoBCBERT, a BERT-based model, utilizing Bio+Clinical BERT as its pre-trained model, and PhenoGPT, a GPT-based model that can be initialized from diverse GPT models, including open-source versions such as GPT-J, Falcon, and LLaMA, as well as closed-source versions such as GPT-3 and GPT-3.5. We compared our methods with PhenoTagger, a recently developed HPO recognition tool that combines rule-based and deep learning methods. We found that our methods can extract more phenotype concepts, including novel ones not characterized by HPO. We also performed case studies on biomedical literature to illustrate how new phenotype information can be recognized and extracted. We compared current BERT-based versus GPT-based models for phenotype tagging, in multiple aspects including model architecture, memory usage, speed, accuracy, and privacy protection. We also discussed the addition of a negation step and an HPO normalization layer to the transformer models for improved HPO term tagging. In conclusion, PhenoBCBERT and PhenoGPT enable the automated discovery of phenotype terms from clinical notes and biomedical literature, facilitating automated downstream tasks to derive new biological insights on human diseases.