# Identifying and Extracting Rare Disease Phenotypes with Large Language Models

arXiv: https://arxiv.org/abs/2306.12656 · HTML: https://mlsanity.com/paper/2306.12656/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 47 references that are in the map.
- First posted on arXiv 22 Jun 2023.

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [Large Language Models are Few-Shot Clinical Information Extractors](https://mlsanity.com/paper/2205.12689/) (2022)
- [Template-Based Named Entity Recognition Using BART](https://mlsanity.com/paper/2106.01760/) (2021)
- [A Unified Generative Framework for Various NER Subtasks](https://mlsanity.com/paper/2106.01223/) (2021)
- [Thinking about GPT-3 In-Context Learning for Biomedical IE? Think Again](https://mlsanity.com/paper/2203.08410/) (2022)
- [Benchmarking large language models for biomedical natural language processing applications and recommendations](https://mlsanity.com/paper/2305.16326/) (2023)
- [Template-free Prompt Tuning for Few-shot NER](https://mlsanity.com/paper/2109.13532/) (2021)
- [LightNER: A Lightweight Tuning Paradigm for Low-resource NER via Pluggable Prompting](https://mlsanity.com/paper/2109.00720/) (2021)
- [Clinical Prompt Learning with Frozen Language Models](https://mlsanity.com/paper/2205.05535/) (2022)
- [The RareDis corpus: a corpus annotated with rare diseases, their signs and symptoms](https://mlsanity.com/paper/2108.01204/) (2021)
- [Exploring deep learning methods for recognizing rare diseases and their clinical manifestations from texts](https://mlsanity.com/paper/2109.00343/) (2021)

## Built on by (most cited first, 18 of 18)

- [Unleashing the potential of prompt engineering for large language models](https://mlsanity.com/paper/2310.14735/) (2023)
- [LongHealth: A Question Answering Benchmark with Long Clinical Documents](https://mlsanity.com/paper/2401.14490/) (2024)
- [Large Language Models in Biomedical and Health Informatics: A Review with Bibliometric Analysis](https://mlsanity.com/paper/2403.16303/) (2024)
- [A Hybrid Framework with Large Language Models for Rare Disease Phenotyping](https://mlsanity.com/paper/2405.10440/) (2024)
- [Large Language Models Struggle in Token-Level Clinical Named Entity Recognition](https://mlsanity.com/paper/2407.00731/) (2024)
- [Surpassing GPT-4 Medical Coding with a Two-Stage Approach](https://mlsanity.com/paper/2311.13735/) (2023)
- [An Evaluation of Large Language Models in Bioinformatics Research](https://mlsanity.com/paper/2402.13714/) (2024)
- [llmNER: (Zero|Few)-Shot Named Entity Recognition, Exploiting the Power of Large Language Models](https://mlsanity.com/paper/2406.04528/) (2024)
- [High-Throughput Phenotyping of Clinical Text Using Large Language Models](https://mlsanity.com/paper/2408.01214/) (2024)
- [Predicting Failures of LLMs to Link Biomedical Ontology Terms to Identifiers Evidence Across Models and Ontologies](https://mlsanity.com/paper/2509.04458/) (2025)
- [Metaphor identification using large language models: A comparison of RAG, prompt engineering, and fine-tuning](https://mlsanity.com/paper/2509.24866/) (2025)
- [Understanding Sarcoidosis Using Large Language Models and Social Media Data](https://mlsanity.com/paper/2405.13005/) (2024)
- [Hybrid Student-Teacher Large Language Model Refinement for Cancer Toxicity Symptom Extraction](https://mlsanity.com/paper/2408.04775/) (2024)
- [Decoding Rarity: Large Language Models in the Diagnosis of Rare Diseases](https://mlsanity.com/paper/2505.17065/) (2025)
- [Leveraging Large Language Models for Rare Disease Named Entity Recognition](https://mlsanity.com/paper/2508.09323/) (2025)
- [Inference Gap in Domain Expertise and Machine Intelligence in Named Entity Recognition: Creation of and Insights from a Substance Use-related Dataset](https://mlsanity.com/paper/2508.19467/) (2025)
- [Beyond Traditional Diagnostics: Transforming Patient-Side Information into Predictive Insights with Knowledge Graphs and Prototypes](https://mlsanity.com/paper/2512.08261/) (2025)
- [An artificial intelligence framework for end-to-end rare disease phenotyping from clinical notes using large language models](https://mlsanity.com/paper/2602.20324/) (2026)

## Abstract (arXiv)

Rare diseases (RDs) are collectively common and affect 300 million people worldwide. Accurate phenotyping is critical for informing diagnosis and treatment, but RD phenotypes are often embedded in unstructured text and time-consuming to extract manually. While natural language processing (NLP) models can perform named entity recognition (NER) to automate extraction, a major bottleneck is the development of a large, annotated corpus for model training. Recently, prompt learning emerged as an NLP paradigm that can lead to more generalizable results without any (zero-shot) or few labeled samples (few-shot). Despite growing interest in ChatGPT, a revolutionary large language model capable of following complex human prompts and generating high-quality responses, none have studied its NER performance for RDs in the zero- and few-shot settings. To this end, we engineered novel prompts aimed at extracting RD phenotypes and, to the best of our knowledge, are the first the establish a benchmark for evaluating ChatGPT's performance in these settings. We compared its performance to the traditional fine-tuning approach and conducted an in-depth error analysis. Overall, fine-tuning BioClinicalBERT resulted in higher performance (F1 of 0.689) than ChatGPT (F1 of 0.472 and 0.591 in the zero- and few-shot settings, respectively). Despite this, ChatGPT achieved similar or higher accuracy for certain entities (i.e., rare diseases and signs) in the one-shot setting (F1 of 0.776 and 0.725). This suggests that with appropriate prompt engineering, ChatGPT has the potential to match or outperform fine-tuned language models for certain entity types with just one labeled sample. While the proliferation of large language models may provide opportunities for supporting RD diagnosis and treatment, researchers and clinicians should critically evaluate model outputs and be well-informed of their limitations.