# Is Multi-Hop Reasoning Really Explainable? Towards Benchmarking Reasoning Interpretability

arXiv: https://arxiv.org/abs/2104.06751 · HTML: https://mlsanity.com/paper/2104.06751/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 27 references that are in the map.
- First posted on arXiv 14 Apr 2021.

## Builds on (most cited first, 14 of 14)

- [Convolutional 2D Knowledge Graph Embeddings](https://mlsanity.com/paper/1707.01476/) (2017)
- [DeepPath: A Reinforcement Learning Method for Knowledge Graph Reasoning](https://mlsanity.com/paper/1707.06690/) (2017)
- [DRUM: End-To-End Differentiable Rule Mining On Knowledge Graphs](https://mlsanity.com/paper/1911.00055/) (2019)
- [End-to-End Differentiable Proving](https://mlsanity.com/paper/1705.11040/) (2017)
- [Go for a Walk and Arrive at the Answer: Reasoning Over Paths in Knowledge Bases using Reinforcement Learning](https://mlsanity.com/paper/1711.05851/) (2017)
- [Multi-Hop Knowledge Graph Reasoning with Reward Shaping](https://mlsanity.com/paper/1808.10568/) (2018)
- [Learning Reasoning Strategies in End-to-End Differentiable Proving](https://mlsanity.com/paper/2007.06477/) (2020)
- [Learning to Explain: Datasets and Models for Identifying Valid Reasoning Chains in Multihop Question-Answering](https://mlsanity.com/paper/2010.03274/) (2020)
- [M-Walk: Learning to Walk over Graphs using Monte Carlo Tree Search](https://mlsanity.com/paper/1802.04394/) (2018)
- [Variational Knowledge Graph Reasoning](https://mlsanity.com/paper/1803.06581/) (2018)
- [Dynamic Anticipation and Completion for Multi-Hop Reasoning over Sparse Knowledge Graph](https://mlsanity.com/paper/2010.01899/) (2020)
- [Collaborative Policy Learning for Open Knowledge Graph Reasoning](https://mlsanity.com/paper/1909.00230/) (2019)
- [Reasoning on Knowledge Graphs with Debate Dynamics](https://mlsanity.com/paper/2001.00461/) (2020)
- [Learning Collaborative Agents with Rule Guidance for Knowledge Graph Reasoning](https://mlsanity.com/paper/2005.00571/) (2020)

## Built on by (most cited first, 10 of 10)

- [Explainable Biomedical Recommendations via Reinforcement Learning Reasoning on Knowledge Graphs](https://mlsanity.com/paper/2111.10625/) (2021)
- [KoRC: Knowledge oriented Reading Comprehension Benchmark for Deep Text Understanding](https://mlsanity.com/paper/2307.03115/) (2023)
- [SQUIRE: A Sequence-to-sequence Framework for Multi-hop Knowledge Graph Reasoning](https://mlsanity.com/paper/2201.06206/) (2022)
- [RGAR: Recurrence Generation-augmented Retrieval for Factual-aware Medical Question Answering](https://mlsanity.com/paper/2502.13361/) (2025)
- [ReOnto: A Neuro-Symbolic Approach for Biomedical Relation Extraction](https://mlsanity.com/paper/2309.01370/) (2023)
- [Untangle the KNOT: Interweaving Conflicting Knowledge and Reasoning Skills in Large Language Models](https://mlsanity.com/paper/2404.03577/) (2024)
- [Large Language Model-Enhanced Symbolic Reasoning for Knowledge Base Completion](https://mlsanity.com/paper/2501.01246/) (2025)
- [Theseus in the Graph: Towards Traceable Multi-Hop Graph Navigation](https://mlsanity.com/paper/2609.14528/) (2026)
- [Emulating the Human Mind: A Neural-symbolic Link Prediction Model with Fast and Slow Reasoning and Filtered Rules](https://mlsanity.com/paper/2310.13996/) (2023)
- [Enhancing Serendipity Recommendation System by Constructing Dynamic User Knowledge Graphs with Large Language Models](https://mlsanity.com/paper/2508.04032/) (2025)

## Abstract (arXiv)

Multi-hop reasoning has been widely studied in recent years to obtain more interpretable link prediction. However, we find in experiments that many paths given by these models are actually unreasonable, while little works have been done on interpretability evaluation for them. In this paper, we propose a unified framework to quantitatively evaluate the interpretability of multi-hop reasoning models so as to advance their development. In specific, we define three metrics including path recall, local interpretability, and global interpretability for evaluation, and design an approximate strategy to calculate them using the interpretability scores of rules. Furthermore, we manually annotate all possible rules and establish a Benchmark to detect the Interpretability of Multi-hop Reasoning (BIMR). In experiments, we run nine baselines on our benchmark. The experimental results show that the interpretability of current multi-hop reasoning models is less satisfactory and is still far from the upper bound given by our benchmark. Moreover, the rule-based models outperform the multi-hop reasoning models in terms of performance and interpretability, which points to a direction for future research, i.e., we should investigate how to better incorporate rule information into the multi-hop reasoning model. Our codes and datasets can be obtained from https://github.com/THU-KEG/BIMR.