# RiTeK: A Dataset for Large Language Models Complex Reasoning over Textual Knowledge Graphs in Medicine

arXiv: https://arxiv.org/abs/2410.13987 · HTML: https://mlsanity.com/paper/2410.13987/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Clinical and biomedical applications, one of 17 directions in LLM reasoning.
- Builds on 18 of its 35 references that are in the map.
- First posted on arXiv 17 Oct 2024.

## Placed in

- LLM reasoning / [Clinical and biomedical applications](https://mlsanity.com/lr/clinical-and-biomedical-applications/)

## Builds on (most cited first, 18 of 18)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [NewsQA: A Machine Comprehension Dataset](https://mlsanity.com/paper/1611.09830/) (2016)
- [Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graph](https://mlsanity.com/paper/2307.07697/) (2023)
- [SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine](https://mlsanity.com/paper/1704.05179/) (2017)
- [G-Retriever: Retrieval-Augmented Generation for Textual Graph Understanding and Question Answering](https://mlsanity.com/paper/2402.07630/) (2024)
- [GNN-RAG: Graph Neural Retrieval for Large Language Model Reasoning](https://mlsanity.com/paper/2405.20139/) (2024)
- [AvaTaR: Optimizing LLM Agents for Tool Usage via Contrastive Reasoning](https://mlsanity.com/paper/2406.11200/) (2024)
- [STaRK: Benchmarking LLM Retrieval on Textual and Relational Knowledge Bases](https://mlsanity.com/paper/2404.13207/) (2024)
- [Graph-constrained Reasoning: Faithful Reasoning on Knowledge Graphs with Large Language Models](https://mlsanity.com/paper/2410.13080/) (2024)
- [Revisiting Summarization Evaluation for Scientific Articles](https://mlsanity.com/paper/1604.00400/) (2016)
- [MedQA-CS: Objective Structured Clinical Examination (OSCE)-Style Benchmark for Evaluating LLM Clinical Skills](https://mlsanity.com/paper/2410.01553/) (2024)
- [BioKGBench: A Knowledge Graph Checking Benchmark of AI Agent for Biomedical Science](https://mlsanity.com/paper/2407.00466/) (2024)
- [Semantic Structure based Query Graph Prediction for Question Answering over Knowledge Graph](https://mlsanity.com/paper/2204.10194/) (2022)

## Built on by (most cited first, 7 of 7)

- [A Survey of Graph Retrieval-Augmented Generation for Customized Large Language Models](https://mlsanity.com/paper/2501.13958/) (2025)
- [RARE: Retrieval-Augmented Reasoning Enhancement for Large Language Models](https://mlsanity.com/paper/2412.02830/) (2024)
- [EvolveRouter: Co-Evolving Routing and Prompt for Multi-Agent Question Answering](https://mlsanity.com/paper/2604.05149/) (2026)
- [AutoGraph-R1: End-to-End Reinforcement Learning for Knowledge Graph Construction](https://mlsanity.com/paper/2510.15339/) (2025)
- [GLEN-Bench: A Graph-Language based Benchmark for Nutritional Health](https://mlsanity.com/paper/2601.18106/) (2026)
- [In-Context Optimization for Retrieval-Augmented Generation: A Gradient-Descent Perspective](https://mlsanity.com/paper/2605.26356/) (2026)
- [When LLM Meets Tree Search: A Systematic View of Inference as Search in Large Language Models](https://mlsanity.com/paper/2608.30395/) (2026)

## Abstract (arXiv)

Answering complex real-world questions in the medical domain often requires accurate retrieval from medical Textual Knowledge Graphs (medical TKGs), as the relational path information from TKGs could enhance the inference ability of Large Language Models (LLMs). However, the main bottlenecks lie in the scarcity of existing medical TKGs, the limited expressiveness of their topological structures, and the lack of comprehensive evaluations of current retrievers for medical TKGs. To address these challenges, we first develop a Dataset1 for LLMs Complex Reasoning over medical Textual Knowledge Graphs (RiTeK), covering a broad range of topological structures. Specifically, we synthesize realistic user queries integrating diverse topological structures, relational information, and complex textual descriptions. We conduct a rigorous medical expert evaluation process to assess and validate the quality of our synthesized queries. RiTeK also serves as a comprehensive benchmark dataset for evaluating the capabilities of retrieval systems built upon LLMs. By assessing 11 representative retrievers on this benchmark, we observe that existing methods struggle to perform well, revealing notable limitations in current LLM-driven retrieval approaches. These findings highlight the pressing need for more effective retrieval systems tailored for semi-structured data in the medical domain.