# An evaluation of GPT models for phenotype concept recognition

arXiv: https://arxiv.org/abs/2309.17169 · HTML: https://mlsanity.com/paper/2309.17169/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 35 references that are in the map.
- First posted on arXiv 29 Sep 2023.

## Builds on (most cited first, 6 of 6)

- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://mlsanity.com/paper/2007.15779/) (2020)
- [BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining](https://mlsanity.com/paper/2210.10341/) (2022)
- [Is GPT-3 a Good Data Annotator?](https://mlsanity.com/paper/2212.10450/) (2022)
- [Benchmarking large language models for biomedical natural language processing applications and recommendations](https://mlsanity.com/paper/2305.16326/) (2023)
- [PhenoTagger: A Hybrid Method for Phenotype Concept Recognition using Human Phenotype Ontology](https://mlsanity.com/paper/2009.08478/) (2020)

## Built on by (most cited first, 10 of 10)

- [AutoPCR: Automated Phenotype Concept Recognition by Prompting](https://mlsanity.com/paper/2507.19315/) (2025)
- [Clinical Insights: A Comprehensive Review of Language Models in Medicine](https://mlsanity.com/paper/2408.11735/) (2024)
- [High-Throughput Phenotyping of Clinical Text Using Large Language Models](https://mlsanity.com/paper/2408.01214/) (2024)
- [Predicting Failures of LLMs to Link Biomedical Ontology Terms to Identifiers Evidence Across Models and Ontologies](https://mlsanity.com/paper/2509.04458/) (2025)
- [Mapping Biomedical Ontology Terms to IDs: Effect of Domain Prevalence on Prediction Accuracy](https://mlsanity.com/paper/2409.13746/) (2024)
- [Leveraging Large Language Models for Rare Disease Named Entity Recognition](https://mlsanity.com/paper/2508.09323/) (2025)
- [On Using Large Language Models to Enhance Clinically-Driven Missing Data Recovery Algorithms in Electronic Health Records](https://mlsanity.com/paper/2510.03844/) (2025)
- [Better Generalizing to Unseen Concepts: An Evaluation Framework and An LLM-Based Auto-Labeled Pipeline for Biomedical Concept Recognition](https://mlsanity.com/paper/2601.16711/) (2026)
- [An artificial intelligence framework for end-to-end rare disease phenotyping from clinical notes using large language models](https://mlsanity.com/paper/2602.20324/) (2026)
- [UMCR: Universal Medical Concept Recognition through Guideline Integration and Agentic Linking](https://mlsanity.com/paper/2610.17363/) (2026)

## Abstract (arXiv)

Objective: Clinical deep phenotyping and phenotype annotation play a critical role in both the diagnosis of patients with rare disorders as well as in building computationally-tractable knowledge in the rare disorders field. These processes rely on using ontology concepts, often from the Human Phenotype Ontology, in conjunction with a phenotype concept recognition task (supported usually by machine learning methods) to curate patient profiles or existing scientific literature. With the significant shift in the use of large language models (LLMs) for most NLP tasks, we examine the performance of the latest Generative Pre-trained Transformer (GPT) models underpinning ChatGPT as a foundation for the tasks of clinical phenotyping and phenotype annotation. Materials and Methods: The experimental setup of the study included seven prompts of various levels of specificity, two GPT models (gpt-3.5-turbo and gpt-4.0) and two established gold standard corpora for phenotype recognition, one consisting of publication abstracts and the other clinical observations. Results: Our results show that, with an appropriate setup, these models can achieve state of the art performance. The best run, using few-shot learning, achieved 0.58 macro F1 score on publication abstracts and 0.75 macro F1 score on clinical observations, the former being comparable with the state of the art, while the latter surpassing the current best in class tool. Conclusion: While the results are promising, the non-deterministic nature of the outcomes, the high cost and the lack of concordance between different runs using the same prompt and input make the use of these LLMs challenging for this particular task.