mlsanityData to 14 Oct 2026

Reasoning on Graphs: Faithful and Interpretable Large Language Model Reasoning

arXiv 2310.01061 ↗First posted 2 Oct 2023cs.CL cs.AIcited by 328

On the map: LLM reasoning → Knowledge- and tool-augmented reasoning

Abstract (arXiv)

Large language models (LLMs) have demonstrated impressive reasoning abilities in complex tasks. However, they lack up-to-date knowledge and experience hallucinations during reasoning, which can lead to incorrect reasoning processes and diminish their performance and trustworthiness. Knowledge graphs (KGs), which capture vast amounts of facts in a structured format, offer a reliable source of knowledge for reasoning. Nevertheless, existing KG-based LLM reasoning methods only treat KGs as factual knowledge bases and overlook the importance of their structural information for reasoning. In this paper, we propose a novel method called reasoning on graphs (RoG) that synergizes LLMs with KGs to enable faithful and interpretable reasoning. Specifically, we present a planning-retrieval-reasoning framework, where RoG first generates relation paths grounded by KGs as faithful plans. These plans are then used to retrieve valid reasoning paths from the KGs for LLMs to conduct faithful reasoning. Furthermore, RoG not only distills knowledge from KGs to improve the reasoning ability of LLMs through training but also allows seamless integration with any arbitrary LLMs during inference. Extensive experiments on two benchmark KGQA datasets demonstrate that RoG achieves state-of-the-art performance on KG reasoning tasks and generates faithful and interpretable reasoning results.

Key facts. Cited by 328 papers in the map (data through 14 Oct 2026). Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning. Builds on 54 of its 76 references that are in the map. First posted on arXiv 2 Oct 2023.

Builds on

54 of 76 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023ChatRule: Mining Logical Rules with Large Language Models for Knowledge Graph Reasoning4 Sep
  2. 2023Graph Reasoning for Question Answering with Triplet Retrieval30 May
  3. 2023Faithful Question Answering with Monte-Carlo Planning4 May
  4. 2022UniKGQA: Unified Retrieval and Reasoning for Solving Multi-hop Question Answering Over Knowledge Graph2 Dec
  5. 2022GreaseLM: Graph REASoning Enhanced Language Models for Question Answering21 Jan

Other references · 49

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  6. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  8. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  10. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  11. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,210
  12. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 894
37 more
  1. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 886
  2. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 668
  3. 2022Towards Reasoning in Large Language Models: A Surveycited by 596
  4. 2023Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Modelscited by 560
  5. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 558
  6. 2018The Web as a Knowledge-base for Answering Complex Questionscited by 410
  7. 2021QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answeringcited by 388
  8. 2016Key-Value Memory Networks for Directly Reading Documentscited by 223
  9. 2019PullNet: Open Domain Question Answering with Iterative Retrieval on Knowledge Bases and Textcited by 184
  10. 2017Variational Reasoning for Question Answering with Knowledge Graphcited by 170
  11. 2018Open Domain Question Answering Using Early Fusion of Knowledge Bases and Textcited by 168
  12. 2022Rethinking with Retrieval: Faithful Large Language Model Inferencecited by 145
  13. 2021End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answeringcited by 133
  14. 2022Faithful Reasoning Using Large Language Modelscited by 130
  15. 2021Improving Multi-hop Knowledge Base Question Answering by Learning Intermediate Supervision Signalscited by 101
  16. 2023Knowledge-Driven CoT: Exploring Faithful Reasoning in LLMs for Knowledge-intensive Question Answeringcited by 97
  17. 2022Subgraph Retrieval Enhanced Model for Multi-hop Knowledge Base Question Answeringcited by 93
  18. 2021RnG-KBQA: Generation Augmented Iterative Ranking for Knowledge Base Question Answeringcited by 86
  19. 2022DecAF: Joint Decoding of Answers and Logical Forms for Question Answering over Knowledge Basescited by 84
  20. 2022Sequence-to-Sequence Knowledge Graph Completion and Question Answeringcited by 75
  21. 2023Structure-free Graph Condensation: From Large-scale Graphs to Condensed Graph-free Datacited by 71
  22. 2022Rethinking and Scaling Up Graph Contrastive Learning: An Extremely Efficient Approach with Group Discriminationcited by 69
  23. 2021KG-FiD: Infusing Knowledge Graph in Fusion-in-Decoder for Open-Domain Question Answeringcited by 67
  24. 2020Relational Message Passing for Knowledge Graph Completioncited by 61
  25. 2021TransferNet: An Effective and Transparent Framework for Multi-hop Question Answering over Relation Graphcited by 60
  26. 2022ArcaneQA: Dynamic Program Induction and Contextualized Encoding for Knowledge Base Question Answeringcited by 48
  27. 2020SPARQA: Skeleton-based Semantic Parsing for Complex Questions over Knowledge Basescited by 47
  28. 2022Knowledge Graph Contrastive Learning Based on Relation-Symmetrical Structurecited by 43
  29. 2022Entailer: Answering Questions with Faithful and Truthful Chains of Reasoningcited by 38
  30. 2022Subgraph Neighboring Relations Infomax for Inductive Link Prediction on Knowledge Graphscited by 27
  31. 2023Learning Strong Graph Neural Networks with Weak Informationcited by 25
  32. 2022Knowledge-in-Context: Towards Knowledgeable Semi-Parametric Language Modelscited by 24
  33. 2023An Empirical Study of Pre-trained Language Models in Simple Knowledge Graph Question Answeringcited by 19
  34. 2023Normalizing Flow-based Neural Process for Few-Shot Knowledge Graph Completioncited by 19
  35. 2023Towards Data-centric Graph Machine Learning: Review and Outlookcited by 15
  36. 2023A Survey on Temporal Knowledge Graph Completion: Taxonomy, Progress, and Prospectscited by 14
  37. 2023Towards Few-shot Inductive Link Prediction on Knowledge Graphs: A Relational Anonymous Walk-guided Neural Process Approachcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 12; 2024: 93; 2025: 144; 2026: 79.

328 papers.

Sort:

By area:

  1. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,212
  2. 2023Time-LLM: Time Series Forecasting by Reprogramming Large Language Modelscited by 700
  3. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 634
  4. 2024Graph Retrieval-Augmented Generation: A Surveycited by 324
  5. 2024HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Modelscited by 281
  6. 2024G-Retriever: Retrieval-Augmented Generation for Textual Graph Understanding and Question Answeringcited by 223
  7. 2023Large Language Models on Graphs: A Comprehensive Surveycited by 192
  8. 2024Retrieval-Augmented Generation with Graphs (GraphRAG)cited by 172
  9. 2024Multi-Step Reasoning with Large Language Models, a Surveycited by 143
  10. 2024GNN-RAG: Graph Neural Retrieval for Large Language Model Reasoningcited by 128
  11. 2024Retrieval-Augmented Generation with Knowledge Graphs for Customer Service Question Answeringcited by 96
  12. 2024KG-Agent: An Efficient Autonomous Agent Framework for Complex Reasoning over Knowledge Graphcited by 93
38 more
  1. 2023Can Knowledge Graphs Reduce Hallucinations in LLMs? : A Surveycited by 92
  2. 2024What Did I Do Wrong? Quantifying LLMs' Sensitivity and Consistency to Prompt Engineeringcited by 90
  3. 2024From Matching to Generation: A Survey on Generative Information Retrievalcited by 89
  4. 2025A Survey of Graph Retrieval-Augmented Generation for Customized Large Language Modelscited by 88
  5. 2025MedRAG: Enhancing Retrieval-augmented Generation with Knowledge Graph-Elicited Reasoning for Healthcare Copilotcited by 83
  6. 2024Simple Is Effective: The Roles of Graphs and Large Language Models in Knowledge-Graph-Based Retrieval-Augmented Generationcited by 80
  7. 2024GRAG: Graph Retrieval-Augmented Generationcited by 78
  8. 2024Plan-on-Graph: Self-Correcting Adaptive Planning of Large Language Model on Knowledge Graphscited by 78
  9. 2024A Survey of Large Language Models for Graphscited by 72
  10. 2024KAG: Boosting LLMs in Professional Domains via Knowledge Augmented Generationcited by 67
  11. 2024Demystifying Chains, Trees, and Graphs of Thoughtscited by 63
  12. 2024Generate-on-Graph: Treat LLM as both Agent and KG in Incomplete Knowledge Graph Question Answeringcited by 60
  13. 2025Knowledge Graph Retrieval-Augmented Generation for LLM-based Recommendationcited by 57
  14. 2024InstructGraph: Boosting Large Language Models via Graph-centric Instruction Tuning and Preference Alignmentcited by 56
  15. 2024KG-RAG: Bridging the Gap Between Knowledge and Creativitycited by 56
  16. 2024Knowledge Mechanisms in Large Language Models: A Survey and Perspectivecited by 51
  17. 2025LLM-based Agents Suffer from Hallucinations: A Survey of Taxonomy, Methods, and Directionscited by 50
  18. 2025Large Language Models Hallucination: A Comprehensive Surveycited by 49
  19. 2025LinearRAG: Linear Graph Retrieval Augmented Generation on Large-scale Corporacited by 49
  20. 2024Graph-constrained Reasoning: Faithful Reasoning on Knowledge Graphs with Large Language Modelscited by 48
  21. 2024Can LLM Graph Reasoning Generalize beyond Pattern Memorization?cited by 47
  22. 2024MegaAgent: A Large-Scale Autonomous LLM-based Multi-Agent System Without Predefined SOPscited by 47
  23. 2024Paths-over-Graph: Knowledge Graph Empowered Large Language Model Reasoningcited by 44
  24. 2024GraphReader: Building Graph-based Agent to Enhance Long-Context Abilities of Large Language Modelscited by 43
  25. 2024Modular RAG: Transforming RAG Systems into LEGO-like Reconfigurable Frameworkscited by 42
  26. 2023Towards Self-Interpretable Graph-Level Anomaly Detectioncited by 41
  27. 2024Large Language Models-guided Dynamic Adaptation for Temporal Knowledge Graph Reasoningcited by 41
  28. 2024On the Prospects of Incorporating Large Language Models (LLMs) in Automated Planning and Scheduling (APS)cited by 40
  29. 2025GFM-RAG: Graph Foundation Model for Retrieval Augmented Generationcited by 40
  30. 2024FiDeLiS: Faithful Reasoning in Large Language Model for Knowledge Graph Question Answeringcited by 39
  31. 2024Multi-hop Question Answering under Temporal Knowledge Editingcited by 36
  32. 2024Graph Machine Learning in the Era of Large Language Models (LLMs)cited by 33
  33. 2024Direct Evaluation of Chain-of-Thought in Multi-hop Reasoning with Knowledge Graphscited by 32
  34. 2023KnowGPT: Knowledge Graph based Prompting for Large Language Modelscited by 31
  35. 2024Knowledge Graphs as Context Sources for LLM-Based Explanations of Learning Recommendationscited by 31
  36. 2024GraphEval: A Knowledge-Graph Based LLM Hallucination Evaluation Frameworkcited by 30
  37. 2024Knowledge Graphs, Large Language Models, and Hallucinations: An NLP Perspectivecited by 29
  38. 2025ArchRAG: Attributed Community-based Hierarchical Retrieval-Augmented Generationcited by 28

50 most cited of 328 All 328 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Grounding LLM Reasoning with Knowledge Graphs
  2. 2023Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graph
  3. 2025PathMind: A Retrieve-Prioritize-Reason Framework for Knowledge Graph Reasoning with Large Language Models
  4. 2024Graph Chain-of-Thought: Augmenting Large Language Models by Reasoning on Graphs
  5. 2024Revisiting the Graph Reasoning Ability of Large Language Models: Case Studies in Translation, Connectivity and Shortest Path
  6. 2023GraphReason: Enhancing Reasoning Capabilities of Large Language Models through A Graph-Based Verification Approach
  7. 2023KG-GPT: A General Framework for Reasoning on Knowledge Graphs Using Large Language Models
  8. 2025KnowPath: Knowledge-enhanced Reasoning via LLM-generated Inference Paths over Knowledge Graphs