# Assessing and Enhancing Large Language Models in Rare Disease Question-answering

arXiv: https://arxiv.org/abs/2408.08422 · HTML: https://mlsanity.com/paper/2408.08422/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 41 references that are in the map.
- First posted on arXiv 15 Aug 2024.

## Builds on (most cited first, 17 of 17)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams](https://mlsanity.com/paper/2009.13081/) (2020)
- [Gemma: Open Models Based on Gemini Research and Technology](https://mlsanity.com/paper/2403.08295/) (2024)
- [Capabilities of GPT-4 on Medical Challenge Problems](https://mlsanity.com/paper/2303.13375/) (2023)
- [MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering](https://mlsanity.com/paper/2203.14371/) (2022)
- [Benchmarking Retrieval-Augmented Generation for Medicine](https://mlsanity.com/paper/2402.13178/) (2024)
- [Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine](https://mlsanity.com/paper/2311.16452/) (2023)
- [BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains](https://mlsanity.com/paper/2402.10373/) (2024)
- [A Comprehensive Evaluation of Large Language Models on Benchmark Biomedical Text Processing Tasks](https://mlsanity.com/paper/2310.04270/) (2023)
- [Large Language Models for Healthcare Data Augmentation: An Example on Patient-Trial Matching](https://mlsanity.com/paper/2303.16756/) (2023)
- [SPeC: A Soft Prompt-Based Calibration on Performance Variability of Large Language Model in Clinical Notes Summarization](https://mlsanity.com/paper/2303.13035/) (2023)
- [Large Language Models Vote: Prompting for Rare Disease Identification](https://mlsanity.com/paper/2308.12890/) (2023)

## Built on by (most cited first, 16 of 16)

- [MAIN-RAG: Multi-Agent Filtering Retrieval-Augmented Generation](https://mlsanity.com/paper/2501.00332/) (2024)
- [Baichuan-M1: Pushing the Medical Capability of Large Language Models](https://mlsanity.com/paper/2502.12671/) (2025)
- [Med-PRM: Medical Reasoning Models with Stepwise, Guideline-verified Process Rewards](https://mlsanity.com/paper/2506.11474/) (2025)
- [Rethinking Diverse Human Preference Learning through Principal Component Analysis](https://mlsanity.com/paper/2502.13131/) (2025)
- [DTS: Enhancing Large Reasoning Models via Decoding Tree Sketching](https://mlsanity.com/paper/2511.00640/) (2025)
- [Unveiling Performance Challenges of Large Language Models in Low-Resource Healthcare: A Demographic Fairness Perspective](https://mlsanity.com/paper/2412.00554/) (2024)
- [QuarkMed Medical Foundation Model Technical Report](https://mlsanity.com/paper/2508.11894/) (2025)
- [Medical Reasoning with Large Language Models: A Survey and MR-Bench](https://mlsanity.com/paper/2604.08559/) (2026)
- [Taylor Unswift: Secured Weight Release for Large Language Models via Taylor Expansion](https://mlsanity.com/paper/2410.05331/) (2024)
- [Fact or Guesswork? Evaluating Large Language Models' Medical Knowledge with Structured One-Hop Judgments](https://mlsanity.com/paper/2502.14275/) (2025)
- [Process Reward Agents for Steering Knowledge-Intensive Reasoning](https://mlsanity.com/paper/2604.09482/) (2026)
- [MMRareBench: A Rare-Disease Multimodal and Multi-Image Medical Benchmark](https://mlsanity.com/paper/2604.10755/) (2026)
- [Eliciting Medical Reasoning with Knowledge-enhanced Data Synthesis: A Semi-Supervised Reinforcement Learning Approach](https://mlsanity.com/paper/2604.11547/) (2026)
- [ChiMed 2.0: Advancing Chinese Medical Dataset in Facilitating Large Language Modeling](https://mlsanity.com/paper/2507.15275/) (2025)
- [Early Risk Prediction with Temporally and Contextually Grounded Clinical Language Processing](https://mlsanity.com/paper/2511.22038/) (2025)
- [From Exposure to Internalization: Dual-Stream Calibration for In-context Clinical Reasoning](https://mlsanity.com/paper/2604.06262/) (2026)

## Abstract (arXiv)

Despite the impressive capabilities of Large Language Models (LLMs) in general medical domains, questions remain about their performance in diagnosing rare diseases. To answer this question, we aim to assess the diagnostic performance of LLMs in rare diseases, and explore methods to enhance their effectiveness in this area. In this work, we introduce a rare disease question-answering (ReDis-QA) dataset to evaluate the performance of LLMs in diagnosing rare diseases. Specifically, we collected 1360 high-quality question-answer pairs within the ReDis-QA dataset, covering 205 rare diseases. Additionally, we annotated meta-data for each question, facilitating the extraction of subsets specific to any given disease and its property. Based on the ReDis-QA dataset, we benchmarked several open-source LLMs, revealing that diagnosing rare diseases remains a significant challenge for these models. To facilitate retrieval augmentation generation for rare disease diagnosis, we collect the first rare diseases corpus (ReCOP), sourced from the National Organization for Rare Disorders (NORD) database. Specifically, we split the report of each rare disease into multiple chunks, each representing a different property of the disease, including their overview, symptoms, causes, effects, related disorders, diagnosis, and standard therapies. This structure ensures that the information within each chunk aligns consistently with a question. Experiment results demonstrate that ReCOP can effectively improve the accuracy of LLMs on the ReDis-QA dataset by an average of 8%. Moreover, it significantly guides LLMs to generate trustworthy answers and explanations that can be traced back to existing literature.