# HOLMES: Hyper-Relational Knowledge Graphs for Multi-hop Question Answering using LLMs

arXiv: https://arxiv.org/abs/2406.06027 · HTML: https://mlsanity.com/paper/2406.06027/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 42 references that are in the map.
- First posted on arXiv 10 Jun 2024.

## Builds on (most cited first, 29 of 29)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering](https://mlsanity.com/paper/2007.01282/) (2020)
- [ConceptNet 5.5: An Open Multilingual Graph of General Knowledge](https://mlsanity.com/paper/1612.03975/) (2016)
- [Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions](https://mlsanity.com/paper/2212.10509/) (2022)
- [COMET: Commonsense Transformers for Automatic Knowledge Graph Construction](https://mlsanity.com/paper/1906.05317/) (2019)
- [StructGPT: A General Framework for Large Language Model to Reason over Structured Data](https://mlsanity.com/paper/2305.09645/) (2023)
- [The Power of Noise: Redefining Retrieval for RAG Systems](https://mlsanity.com/paper/2401.14887/) (2024)
- [Sentiment Analysis in the Era of Large Language Models: A Reality Check](https://mlsanity.com/paper/2305.15005/) (2023)
- [Leveraging Large Language Models for Multiple Choice Question Answering](https://mlsanity.com/paper/2210.12353/) (2022)
- [Large Language Models for Generative Information Extraction: A Survey](https://mlsanity.com/paper/2312.17617/) (2023)
- [PullNet: Open Domain Question Answering with Iterative Retrieval on Knowledge Bases and Text](https://mlsanity.com/paper/1904.09537/) (2019)
- [Open Domain Question Answering Using Early Fusion of Knowledge Bases and Text](https://mlsanity.com/paper/1809.00782/) (2018)
- [GPT-NER: Named Entity Recognition via Large Language Models](https://mlsanity.com/paper/2304.10428/) (2023)
- [Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answering](https://mlsanity.com/paper/1911.10470/) (2019)
- [Unsupervised Question Decomposition for Question Answering](https://mlsanity.com/paper/2002.09758/) (2020)
- [Revisiting Relation Extraction in the era of Large Language Models](https://mlsanity.com/paper/2305.05003/) (2023)
- [Hierarchical Graph Network for Multi-hop Question Answering](https://mlsanity.com/paper/1911.03631/) (2019)
- [Dynamically Fused Graph Network for Multi-hop Reasoning](https://mlsanity.com/paper/1905.06933/) (2019)
- [Answering Open-Domain Questions of Varying Reasoning Steps from Text](https://mlsanity.com/paper/2010.12527/) (2020)
- [Leveraging Structured Information for Explainable Multi-hop Question Answering and Reasoning](https://mlsanity.com/paper/2311.03734/) (2023)
- [Modeling Multi-hop Question Answering as Single Sequence Prediction](https://mlsanity.com/paper/2205.09226/) (2022)
- [Interpretable AMR-Based Question Decomposition for Multi-hop Question Answering](https://mlsanity.com/paper/2206.08486/) (2022)
- [Gemini Pro Defeated by GPT-4V: Evidence from Education](https://mlsanity.com/paper/2401.08660/) (2023)
- [Iterative Hierarchical Attention for Answering Complex Questions over Long Documents](https://mlsanity.com/paper/2106.00200/) (2021)
- [Single Sequence Prediction over Reasoning Graphs for Multi-hop QA](https://mlsanity.com/paper/2307.00335/) (2023)
- [Question Answering from Unstructured Text by Retrieval and Comprehension](https://mlsanity.com/paper/1703.08885/) (2017)

## Built on by (most cited first, 24 of 24)

- [AriGraph: Learning Knowledge Graph World Models with Episodic Memory for LLM Agents](https://mlsanity.com/paper/2407.04363/) (2024)
- [A Survey of Graph Retrieval-Augmented Generation for Customized Large Language Models](https://mlsanity.com/paper/2501.13958/) (2025)
- [StructRAG: Boosting Knowledge Intensive Reasoning of LLMs via Inference-time Hybrid Information Structurization](https://mlsanity.com/paper/2410.08815/) (2024)
- [PathRAG: Pruning Graph-based Retrieval Augmented Generation with Relational Paths](https://mlsanity.com/paper/2502.14902/) (2025)
- [GFM-RAG: Graph Foundation Model for Retrieval Augmented Generation](https://mlsanity.com/paper/2502.01113/) (2025)
- [Large Language Models Meet Knowledge Graphs for Question Answering: Synthesis and Opportunities](https://mlsanity.com/paper/2505.20099/) (2025)
- [GIVE: Structured Reasoning of Large Language Models with Knowledge Graph Inspired Veracity Extrapolation](https://mlsanity.com/paper/2410.08475/) (2024)
- [SRAG: Structured Retrieval-Augmented Generation for Multi-Entity Question Answering over Wikipedia Graph](https://mlsanity.com/paper/2503.01346/) (2025)
- [Memory-augmented Query Reconstruction for LLM-based Knowledge Graph Reasoning](https://mlsanity.com/paper/2503.05193/) (2025)
- [SuperRAG: Beyond RAG with Layout-Aware Graph Modeling](https://mlsanity.com/paper/2503.04790/) (2025)
- [Loki's Dance of Illusions: A Comprehensive Survey of Hallucination in Large Language Models](https://mlsanity.com/paper/2507.02870/) (2025)
- [Enrich-on-Graph: Query-Graph Alignment for Complex Reasoning with LLM Enriching](https://mlsanity.com/paper/2509.20810/) (2025)
- [Query-Aware Flow Diffusion for Graph-Based RAG with Retrieval Guarantees](https://mlsanity.com/paper/2605.18775/) (2026)
- [How to Mitigate Information Loss in Knowledge Graphs for GraphRAG: Leveraging Triple Context Restoration and Query-Driven Feedback](https://mlsanity.com/paper/2501.15378/) (2025)
- [Wikontic: Constructing Wikidata-Aligned, Ontology-Aware Knowledge Graphs with Large Language Models](https://mlsanity.com/paper/2512.00590/) (2025)
- [Relink: Constructing Query-Driven Evidence Graph On-the-Fly for GraphRAG](https://mlsanity.com/paper/2601.07192/) (2026)
- [Cross-Granularity Hypergraph Retrieval-Augmented Generation for Multi-hop Question Answering](https://mlsanity.com/paper/2508.11247/) (2025)
- [DAVIS: Planning Agent with Knowledge Graph-Powered Inner Monologue](https://mlsanity.com/paper/2410.09252/) (2024)
- [Large Language Models for Knowledge Graph Embedding: A Survey](https://mlsanity.com/paper/2501.07766/) (2025)
- [KCS: Diversify Multi-hop Question Generation with Knowledge Composition Sampling](https://mlsanity.com/paper/2508.20567/) (2025)
- [CompoST: A Benchmark for Analyzing the Ability of LLMs To Compositionally Interpret Questions in a QALD Setting](https://mlsanity.com/paper/2507.21257/) (2025)
- [HELIOS: Harmonizing Early Fusion, Late Fusion, and LLM Reasoning for Multi-Granular Table-Text Retrieval](https://mlsanity.com/paper/2603.02248/) (2026)
- [Self-supervised Attribute-aware Dynamic Preference Ranking Alignment](https://mlsanity.com/paper/2502.12189/) (2025)
- [BRIT: Bidirectional Retrieval over Unified Image-Text Graph](https://mlsanity.com/paper/2505.18450/) (2025)

## Abstract (arXiv)

Given unstructured text, Large Language Models (LLMs) are adept at answering simple (single-hop) questions. However, as the complexity of the questions increase, the performance of LLMs degrade. We believe this is due to the overhead associated with understanding the complex question followed by filtering and aggregating unstructured information in the raw text. Recent methods try to reduce this burden by integrating structured knowledge triples into the raw text, aiming to provide a structured overview that simplifies information processing. However, this simplistic approach is query-agnostic and the extracted facts are ambiguous as they lack context. To address these drawbacks and to enable LLMs to answer complex (multi-hop) questions with ease, we propose to use a knowledge graph (KG) that is context-aware and is distilled to contain query-relevant information. The use of our compressed distilled KG as input to the LLM results in our method utilizing up to 67% fewer tokens to represent the query relevant information present in the supporting documents, compared to the state-of-the-art (SoTA) method. Our experiments show consistent improvements over the SoTA across several metrics (EM, F1, BERTScore, and Human Eval) on two popular benchmark datasets (HotpotQA and MuSiQue).