# Leveraging Large Language Models for Rare Disease Named Entity Recognition

arXiv: https://arxiv.org/abs/2508.09323 · HTML: https://mlsanity.com/paper/2508.09323/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 30 of its 55 references that are in the map.
- First posted on arXiv 12 Aug 2025.

## Builds on (most cited first, 30 of 30)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [Large Language Models are Few-Shot Clinical Information Extractors](https://mlsanity.com/paper/2205.12689/) (2022)
- [Benchmarking large language models for biomedical natural language processing applications and recommendations](https://mlsanity.com/paper/2305.16326/) (2023)
- [Multi-domain Clinical Natural Language Processing with MedCAT: the Medical Concept Annotation Toolkit](https://mlsanity.com/paper/2010.01165/) (2020)
- [Identifying and Extracting Rare Disease Phenotypes with Large Language Models](https://mlsanity.com/paper/2306.12656/) (2023)
- [SummQA at MEDIQA-Chat 2023:In-Context Learning with GPT-4 for Medical Summarization](https://mlsanity.com/paper/2306.17384/) (2023)
- [De-identification of clinical free text using natural language processing: A systematic review of current approaches](https://mlsanity.com/paper/2312.03736/) (2023)
- [Few-shot clinical entity recognition in English, French and Spanish: masked language models outperform generative model prompting](https://mlsanity.com/paper/2402.12801/) (2024)
- [The RareDis corpus: a corpus annotated with rare diseases, their signs and symptoms](https://mlsanity.com/paper/2108.01204/) (2021)
- [An evaluation of GPT models for phenotype concept recognition](https://mlsanity.com/paper/2309.17169/) (2023)
- [Large Language Models Vote: Prompting for Rare Disease Identification](https://mlsanity.com/paper/2308.12890/) (2023)
- [Generalizing over Long Tail Concepts for Medical Term Normalization](https://mlsanity.com/paper/2210.11947/) (2022)
- [Exploring deep learning methods for recognizing rare diseases and their clinical manifestations from texts](https://mlsanity.com/paper/2109.00343/) (2021)
- [A Hybrid Framework with Large Language Models for Rare Disease Phenotyping](https://mlsanity.com/paper/2405.10440/) (2024)
- [Large Language Models Struggle in Token-Level Clinical Named Entity Recognition](https://mlsanity.com/paper/2407.00731/) (2024)
- [Orthogonal Subsampling for Big Data Linear Regression](https://mlsanity.com/paper/2105.14647/) (2021)
- [Comprehensive and Practical Evaluation of Retrieval-Augmented Generation Systems for Medical Question Answering](https://mlsanity.com/paper/2411.09213/) (2024)
- [Improving The Diagnosis of Thyroid Cancer by Machine Learning and Clinical Data](https://mlsanity.com/paper/2203.15804/) (2022)
- [A Fast Bootstrap Algorithm for Causal Inference with Large Data](https://mlsanity.com/paper/2302.02859/) (2023)
- [Independence-Encouraging Subsampling for Nonparametric Additive Models](https://mlsanity.com/paper/2302.13441/) (2023)
- [Understanding Sarcoidosis Using Large Language Models and Social Media Data](https://mlsanity.com/paper/2405.13005/) (2024)
- [Efficient subsampling for high-dimensional data](https://mlsanity.com/paper/2411.06298/) (2024)

## Abstract (arXiv)

Named Entity Recognition (NER) in the rare disease domain poses unique challenges due to limited labeled data, semantic ambiguity between entity types, and long-tail distributions. In this study, we evaluate the capabilities of GPT-4o for rare disease NER under low-resource settings, using a range of prompt-based strategies including zero-shot prompting, few-shot in-context learning, retrieval-augmented generation (RAG), and task-level fine-tuning. We design a structured prompting framework that encodes domain-specific knowledge and disambiguation rules for four entity types. We further introduce two semantically guided few-shot example selection methods to improve in-context performance while reducing labeling effort. Experiments on the RareDis Corpus show that GPT-4o achieves competitive or superior performance compared to BioClinicalBERT, with task-level fine-tuning yielding the strongest performance among the evaluated approaches and improving upon the previously reported BioClinicalBERT baseline. Cost-performance analysis reveals that few-shot prompting delivers high returns at low token budgets. RAG provides limited overall gains but can improve recall for challenging entity types, especially signs and symptoms. An error taxonomy highlights common failure modes such as boundary drift and type confusion, suggesting opportunities for post-processing and hybrid refinement. Our results demonstrate that prompt-optimized LLMs can serve as effective, scalable alternatives to traditional supervised models in biomedical NER, particularly in rare disease applications where annotated data is scarce.