# Integrating UMLS Knowledge into Large Language Models for Medical Question Answering

arXiv: https://arxiv.org/abs/2310.02778 · HTML: https://mlsanity.com/paper/2310.02778/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 23 references that are in the map.
- First posted on arXiv 4 Oct 2023.

## Builds on (most cited first, 9 of 9)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Towards Expert-Level Medical Question Answering with Large Language Models](https://mlsanity.com/paper/2305.09617/) (2023)
- [Almanac: Retrieval-Augmented Language Models for Clinical Medicine](https://mlsanity.com/paper/2303.01229/) (2023)
- [ExpertQA: Expert-Curated Questions and Attributed Answers](https://mlsanity.com/paper/2309.07852/) (2023)
- [Neural Natural Language Processing for Unstructured Data in Electronic Health Records: a Review](https://mlsanity.com/paper/2107.02975/) (2021)
- [Integrating Graph Contextualized Knowledge into Pre-trained Language Models](https://mlsanity.com/paper/1912.00147/) (2019)
- [Unsupervised Cross-Domain Prerequisite Chain Learning using Variational Graph Autoencoders](https://mlsanity.com/paper/2105.03505/) (2021)
- [EHRKit: A Python Natural Language Processing Toolkit for Electronic Health Record Texts](https://mlsanity.com/paper/2204.06604/) (2022)

## Built on by (most cited first, 10 of 10)

- [Development and Testing of Retrieval Augmented Generation in Large Language Models -- A Case Study Report](https://mlsanity.com/paper/2402.01733/) (2024)
- [Towards Omni-RAG: Comprehensive Retrieval-Augmented Generation for Large Language Models in Medical Applications](https://mlsanity.com/paper/2501.02460/) (2025)
- [A Case Study Exploring the Current Landscape of Synthetic Medical Record Generation with Commercial LLMs](https://mlsanity.com/paper/2504.14657/) (2025)
- [Generalizable and Scalable Multistage Biomedical Concept Normalization Leveraging Large Language Models](https://mlsanity.com/paper/2405.15122/) (2024)
- [Rethinking Retrieval-Augmented Generation for Medicine: A Large-Scale, Systematic Expert Evaluation and Practical Insights](https://mlsanity.com/paper/2511.06738/) (2025)
- [Development of a Reliable and Accessible Caregiving Language Model (CaLM)](https://mlsanity.com/paper/2403.06857/) (2024)
- [Ascle: A Python Natural Language Processing Toolkit for Medical Text Generation](https://mlsanity.com/paper/2311.16588/) (2023)
- [Leveraging Large Language Models for Concept Graph Recovery and Question Answering in NLP Education](https://mlsanity.com/paper/2402.14293/) (2024)
- [AlzheimerRAG: Multimodal Retrieval Augmented Generation for Clinical Use Cases using PubMed articles](https://mlsanity.com/paper/2412.16701/) (2024)
- [oRetrieval Augmented Generation for 10 Large Language Models and its Generalizability in Assessing Medical Fitness](https://mlsanity.com/paper/2410.08431/) (2024)

## Abstract (arXiv)

Large language models (LLMs) have demonstrated powerful text generation capabilities, bringing unprecedented innovation to the healthcare field. While LLMs hold immense promise for applications in healthcare, applying them to real clinical scenarios presents significant challenges, as these models may generate content that deviates from established medical facts and even exhibit potential biases. In our research, we develop an augmented LLM framework based on the Unified Medical Language System (UMLS), aiming to better serve the healthcare community. We employ LLaMa2-13b-chat and ChatGPT-3.5 as our benchmark models, and conduct automatic evaluations using the ROUGE Score and BERTScore on 104 questions from the LiveQA test set. Additionally, we establish criteria for physician-evaluation based on four dimensions: Factuality, Completeness, Readability and Relevancy. ChatGPT-3.5 is used for physician evaluation with 20 questions on the LiveQA test set. Multiple resident physicians conducted blind reviews to evaluate the generated content, and the results indicate that this framework effectively enhances the factuality, completeness, and relevance of generated content. Our research demonstrates the effectiveness of using UMLS-augmented LLMs and highlights the potential application value of LLMs in in medical question-answering.