# KG-Rank: Enhancing Large Language Models for Medical QA with Knowledge Graphs and Ranking Techniques

arXiv: https://arxiv.org/abs/2403.05881 · HTML: https://mlsanity.com/paper/2403.05881/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 26 references that are in the map.
- First posted on arXiv 9 Mar 2024.

## Builds on (most cited first, 12 of 12)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance](https://mlsanity.com/paper/1909.02622/) (2019)
- [MedCPT: Contrastive Pre-trained Transformers with Large-scale PubMed Search Logs for Zero-shot Biomedical Information Retrieval](https://mlsanity.com/paper/2307.00589/) (2023)
- [UmlsBERT: Clinical Domain Knowledge Augmentation of Contextual Embeddings Using the Unified Medical Language System Metathesaurus](https://mlsanity.com/paper/2010.10391/) (2020)
- [Neural Natural Language Processing for Unstructured Data in Electronic Health Records: a Review](https://mlsanity.com/paper/2107.02975/) (2021)
- [Enhancing Diagnostic Accuracy through Multi-Agent Conversations: Using Large Language Models to Mitigate Cognitive Bias](https://mlsanity.com/paper/2401.14589/) (2024)
- [Diffuser: Efficient Transformers with Multi-hop Attention Diffusion for Long Sequences](https://mlsanity.com/paper/2210.11794/) (2022)
- [Going Beyond Local: Global Graph-Enhanced Personalized News Recommendations](https://mlsanity.com/paper/2307.06576/) (2023)
- [Leveraging Large Language Models for Concept Graph Recovery and Question Answering in NLP Education](https://mlsanity.com/paper/2402.14293/) (2024)

## Built on by (most cited first, 30 of 37)

- [Graph Retrieval-Augmented Generation: A Survey](https://mlsanity.com/paper/2408.08921/) (2024)
- [Retrieval-Augmented Generation with Graphs (GraphRAG)](https://mlsanity.com/paper/2501.00309/) (2024)
- [Context-DPO: Aligning Language Models for Context-Faithfulness](https://mlsanity.com/paper/2412.15280/) (2024)
- [Graph-constrained Reasoning: Faithful Reasoning on Knowledge Graphs with Large Language Models](https://mlsanity.com/paper/2410.13080/) (2024)
- [Decoding by Contrasting Knowledge: Enhancing LLMs' Confidence on Edited Facts](https://mlsanity.com/paper/2405.11613/) (2024)
- [A Survey on Medical Large Language Models: Technology, Application, Trustworthiness, and Future Directions](https://mlsanity.com/paper/2406.03712/) (2024)
- [Large Language Models Meet Knowledge Graphs for Question Answering: Synthesis and Opportunities](https://mlsanity.com/paper/2505.20099/) (2025)
- [A Benchmark for Long-Form Medical Question Answering](https://mlsanity.com/paper/2411.09834/) (2024)
- [KGARevion: An AI Agent for Knowledge-Intensive Biomedical QA](https://mlsanity.com/paper/2410.04660/) (2024)
- [RAG-Enhanced Collaborative LLM Agents for Drug Discovery](https://mlsanity.com/paper/2502.17506/) (2025)
- [Graph-based Approaches and Functionalities in Retrieval-Augmented Generation: A Comprehensive Survey](https://mlsanity.com/paper/2504.10499/) (2025)
- [medIKAL: Integrating Knowledge Graphs as Assistants of LLMs for Enhanced Clinical Diagnosis on EMRs](https://mlsanity.com/paper/2406.14326/) (2024)
- [Towards Omni-RAG: Comprehensive Retrieval-Augmented Generation for Large Language Models in Medical Applications](https://mlsanity.com/paper/2501.02460/) (2025)
- [Agentic Medical Knowledge Graphs Enhance Medical Question Answering: Bridging the Gap Between LLMs and Evolving Medical Knowledge](https://mlsanity.com/paper/2502.13010/) (2025)
- [GraphCheck: Breaking Long-Term Text Barriers with Extracted Knowledge Graph-Powered Fact-Checking](https://mlsanity.com/paper/2502.16514/) (2025)
- [GraphRAG-R1: Graph Retrieval-Augmented Generation with Process-Constrained Reinforcement Learning](https://mlsanity.com/paper/2507.23581/) (2025)
- [AGENTiGraph: A Multi-Agent Knowledge Graph Framework for Interactive, Domain-Specific LLM Chatbots](https://mlsanity.com/paper/2508.02999/) (2025)
- [Graphusion: Leveraging Large Language Models for Scientific Knowledge Graph Fusion and Construction in NLP Education](https://mlsanity.com/paper/2407.10794/) (2024)
- [AGENTiGraph: An Interactive Knowledge Graph Platform for LLM-based Chatbots Utilizing Private Data](https://mlsanity.com/paper/2410.11531/) (2024)
- [Retrieval-Augmented Generation in Medicine: A Scoping Review of Technical Implementations, Clinical Applications, and Ethical Considerations](https://mlsanity.com/paper/2511.05901/) (2025)
- [Retrieval-Augmented Generation for Generative Artificial Intelligence in Medicine](https://mlsanity.com/paper/2406.12449/) (2024)
- [Large Language Models for Knowledge Graph Embedding: A Survey](https://mlsanity.com/paper/2501.07766/) (2025)
- [DALL-M: Context-Aware Clinical Data Augmentation with LLMs](https://mlsanity.com/paper/2407.08227/) (2024)
- [Fact or Guesswork? Evaluating Large Language Models' Medical Knowledge with Structured One-Hop Judgments](https://mlsanity.com/paper/2502.14275/) (2025)
- [Toward Revealing Nuanced Biases in Medical LLMs](https://mlsanity.com/paper/2507.21176/) (2025)
- [HeteroRAG: A Heterogeneous Retrieval-Augmented Generation Framework for Medical Vision Language Tasks](https://mlsanity.com/paper/2508.12778/) (2025)
- [MED-COPILOT: A Medical Assistant Powered by GraphRAG and Similar Patient Case Retrieval](https://mlsanity.com/paper/2603.00460/) (2026)
- [Exploring the Role of Knowledge Graph-Based RAG in Japanese Medical Question Answering with Small-Scale LLMs](https://mlsanity.com/paper/2504.10982/) (2025)
- [The Evolving Landscape of Generative Large Language Models and Traditional Natural Language Processing in Medicine](https://mlsanity.com/paper/2505.10261/) (2025)
- [SlimRAG: Retrieval without Graphs via Entity-Aware Context Selection](https://mlsanity.com/paper/2506.17288/) (2025)

## Abstract (arXiv)

Large language models (LLMs) have demonstrated impressive generative capabilities with the potential to innovate in medicine. However, the application of LLMs in real clinical settings remains challenging due to the lack of factual consistency in the generated content. In this work, we develop an augmented LLM framework, KG-Rank, which leverages a medical knowledge graph (KG) along with ranking and re-ranking techniques, to improve the factuality of long-form question answering (QA) in the medical domain. Specifically, when receiving a question, KG-Rank automatically identifies medical entities within the question and retrieves the related triples from the medical KG to gather factual information. Subsequently, KG-Rank innovatively applies multiple ranking techniques to refine the ordering of these triples, providing more relevant and precise information for LLM inference. To the best of our knowledge, KG-Rank is the first application of KG combined with ranking models in medical QA specifically for generating long answers. Evaluation on four selected medical QA datasets demonstrates that KG-Rank achieves an improvement of over 18% in ROUGE-L score. Additionally, we extend KG-Rank to open domains, including law, business, music, and history, where it realizes a 14% improvement in ROUGE-L score, indicating the effectiveness and great potential of KG-Rank.