mlsanityData to 7 Oct 2026

FiDeLiS: Faithful Reasoning in Large Language Model for Knowledge Graph Question Answering

arXiv 2405.13873 ↗First posted 22 May 2024cs.AI cs.CLcited by 38

On the map: LLM reasoning → Knowledge- and tool-augmented reasoning

Abstract (arXiv)

Large Language Models (LLMs) are often challenged by generating erroneous or hallucinated responses, especially in complex reasoning tasks. Leveraging Knowledge Graphs (KGs) as external knowledge sources has emerged as a viable solution. However, existing KG-enhanced methods, either retrieval-based or agent-based, encounter difficulties in accurately retrieving knowledge and efficiently traversing KGs at scale. In this paper, we propose a unified framework, FiDeLiS, designed to improve the factuality of LLM responses by anchoring answers to verifiable reasoning steps retrieved from KGs. To achieve this, we leverage step-wise beam search with a deductive scoring function, allowing the LLM to validate reasoning process step by step, and halt the search once the question is deducible. In addition, we propose a Path-RAG module to pre-select a smaller candidate set for each beam search step, reducing computational costs by narrowing the search space. Extensive experiments show that our method, as a training-free framework, not only improve the performance but also enhance the factuality and interpretability across different benchmarks. Code is released at https://github.com/Y-Sui/FiDeLiS.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning. Builds on 53 of its 81 references that are in the map. First posted on arXiv 22 May 2024.

Builds on

53 of 81 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024CR-LT-KGQA: A Knowledge Graph Question Answering Dataset Requiring Commonsense Reasoning and Long-Tail Knowledge3 Mar
  2. 2023Reasoning on Graphs: Faithful and Interpretable Large Language Model Reasoning2 Oct
  3. 2023Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graph15 Jul
  4. 2023Graph Reasoning for Question Answering with Triplet Retrieval30 May
  5. 2023Faithful Question Answering with Monte-Carlo Planning4 May
  6. 2022UniKGQA: Unified Retrieval and Reasoning for Solving Multi-hop Question Answering Over Knowledge Graph2 Dec
  7. 2022Large Language Models are few(1)-shot Table Reasoners13 Oct

Other references · 46

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  8. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  9. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  10. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  11. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  12. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
34 more
  1. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  2. 2022Galactica: A Large Language Model for Sciencecited by 736
  3. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 663
  4. 2022Towards Reasoning in Large Language Models: A Surveycited by 593
  5. 2023Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Modelscited by 550
  6. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 459
  7. 2018The Web as a Knowledge-base for Answering Complex Questionscited by 409
  8. 2021QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answeringcited by 386
  9. 2023StructGPT: A General Framework for Large Language Model to Reason over Structured Datacited by 339
  10. 2024Hallucination is Inevitable: An Innate Limitation of Large Language Modelscited by 329
  11. 2022Binding Language Models in Symbolic Languagescited by 224
  12. 2016Key-Value Memory Networks for Directly Reading Documentscited by 223
  13. 2023Self-Evaluation Guided Beam Search for Reasoningcited by 195
  14. 2019PullNet: Open Domain Question Answering with Iterative Retrieval on Knowledge Bases and Textcited by 182
  15. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 162
  16. 2022Rethinking with Retrieval: Faithful Large Language Model Inferencecited by 145
  17. 2023Deductive Verification of Chain-of-Thought Reasoningcited by 122
  18. 2023Knowledge-Augmented Language Model Prompting for Zero-Shot Knowledge Graph Question Answeringcited by 119
  19. 2021Case-based Reasoning for Natural Language Queries over Knowledge Basescited by 107
  20. 2021Improving Multi-hop Knowledge Base Question Answering by Learning Intermediate Supervision Signalscited by 100
  21. 2023Knowledge-Driven CoT: Exploring Faithful Reasoning in LLMs for Knowledge-intensive Question Answeringcited by 97
  22. 2023Can Knowledge Graphs Reduce Hallucinations in LLMs? : A Surveycited by 92
  23. 2024The Dawn After the Dark: An Empirical Study on Factuality Hallucination in Large Language Modelscited by 87
  24. 2022DecAF: Joint Decoding of Answers and Logical Forms for Question Answering over Knowledge Basescited by 83
  25. 2024Scientific Large Language Models: A Survey on Biological & Chemical Domainscited by 80
  26. 2024RAP: Retrieval-Augmented Planning with Contextual Memory for Multimodal LLM Agentscited by 80
  27. 2022A Survey of Knowledge Enhanced Pre-trained Language Modelscited by 76
  28. 2022Don't Generate, Discriminate: A Proposal for Grounding Language Models to Real-World Environmentscited by 57
  29. 2022ArcaneQA: Dynamic Program Induction and Contextualized Encoding for Knowledge Base Question Answeringcited by 48
  30. 2020SPARQA: Skeleton-based Semantic Parsing for Complex Questions over Knowledge Basescited by 47
  31. 2023ChatKBQA: A Generate-then-Retrieve Framework for Knowledge Base Question Answering with Fine-tuned Large Language Modelscited by 44
  32. 2023Thought Propagation: An Analogical Approach to Complex Reasoning with Large Language Modelscited by 37
  33. 2023Generate, Transform, Answer: Question Specific Tool Synthesis for Tabular Datacited by 11
  34. 2018When to Finish? Optimal Beam Search for Neural Text Generation (modulo beam size)cited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 17; 2026: 15.

38 papers.

Sort:

By area:

  1. 2025Efficient Inference for Large Reasoning Models: A Surveycited by 58
  2. 2024Graph-constrained Reasoning: Faithful Reasoning on Knowledge Graphs with Large Language Modelscited by 47
  3. 2025UniGraph2: Learning a Unified Embedding Space to Bind Multimodal Graphscited by 47
  4. 2025EvoTest: Evolutionary Test-Time Learning for Self-Improving Agentic Systemscited by 34
  5. 2025Meta-Reasoner: Dynamic Guidance for Optimized Inference-time Reasoning in Large Language Modelscited by 33
  6. 2025Evaluating the Paperclip Maximizer: Are RL-Based Language Models More Likely to Pursue Instrumental Goals?cited by 26
  7. 2024Harnessing Large Language Models for Knowledge Graph Question Answering via Adaptive Multi-Aspect Retrieval-Augmentationcited by 24
  8. 2024Can Knowledge Graphs Make Large Language Models More Trustworthy? An Empirical Study Over Open-ended Question Answeringcited by 20
  9. 2025Enabling Self-Improving Agents to Learn at Test Time With Human-In-The-Loop Guidancecited by 14
  10. 2025Exploring Graph Learning Tasks with Pure LLMs: A Comprehensive Benchmark and Investigationcited by 11
  11. 2025BYOKG-RAG: Multi-Strategy Graph Retrieval for Knowledge Graph Question Answeringcited by 10
  12. 2025Align-GRAG: Anchor and Rationale Guided Dual Alignment for Graph Retrieval-Augmented Generationcited by 9
26 more
  1. 2024Chain-of-Probe: Examining the Necessity and Accuracy of CoT Step-by-Stepcited by 8
  2. 2025Beyond Seen Data: Improving KBQA Generalization Through Schema-Guided Logical Form Generationcited by 8
  3. 2025ReMindRAG: Low-Cost LLM-Guided Knowledge Graph Traversal for Efficient RAGcited by 8
  4. 2026EvoClinician: A Self-Evolving Agent for Multi-Turn Medical Diagnosis via Test-Time Evolutionary Learningcited by 8
  5. 2024Generalizing Graph Transformers Across Diverse Graphs and Tasks via Pre-trainingcited by 7
  6. 2025DAMR: Efficient and Adaptive Context-Aware Knowledge Graph Question Answering with LLM-Guided MCTScited by 6
  7. 2025GraphRAFT: Retrieval Augmented Fine-Tuning for Knowledge Graphs on Graph Databasescited by 5
  8. 2025Search-on-Graph: Iterative Informed Navigation for Large Language Model Reasoning on Knowledge Graphscited by 5
  9. 2025Advancing Multi-Agent RAG Systems with Minimalist Reinforcement Learningcited by 3
  10. 2025Beyond Black-Box Interventions: Latent Probing for Faithful Retrieval-Augmented Generationcited by 2
  11. 2026Search-on-Graph-R1: Training Large Language Models to Search Knowledge Graphs with Reinforcement Learningcited by 2
  12. 2024Semantic Parsing with Candidate Expressions for Knowledge Base Question Answeringcited by 1
  13. 2025Encoder-Free Knowledge-Graph Reasoning with LLMs via Hyperdimensional Path Retrievalcited by 1
  14. 2026Structured Semantic Information Helps Retrieve Better Examples for In-Context Learning Applied to Few-Shot Relation Extractioncited by 1
  15. 2026RouterKGQA: Specialized--General Model Routing for Constraint-Aware Knowledge Graph Question Answeringcited by 1
  16. 2026The Scaling Properties of Implicit Deductive Reasoning in Transformerscited by 1
  17. 2026Towards general embodied intelligence: integrating large language models, knowledge bases, and reasoning capabilities to build the next generation of AI agentscited by 1
  18. 2026The EpisTwin: A Knowledge Graph-Grounded Neuro-Symbolic Architecture for Personal AIcited by 0
  19. 2026TRACE: An Experiential Framework for Coherent Multi-hop Knowledge Graph Question Answeringcited by 0
  20. 2026STEM: Structure-Tracing Evidence Mining for Knowledge Graphs-Driven Retrieval-Augmented Generationcited by 0
  21. 2026Reward on Path: Learning Intermediate Supervision Signals for Knowledge Graph Question Answeringcited by 0
  22. 2026BadSKP: Backdoor Attacks on Knowledge Graph-Enhanced LLMs with Soft Promptscited by 0
  23. 2026Faithful by Definition: Emotion Analysis via Natural Semantic Metalanguage Explicationscited by 0
  24. 2026Mapping Text to Multiplex Graph: Prompt Compression as L'evy Walk-Guided Graph Pruningcited by 0
  25. 2026Query-Side Attacks on GNN-Based KGQA: Tracing Failures from Entity Linking to Answer Generationcited by 0
  26. 2026The Path Matters: Evaluating Small Language Models Beyond Answer Accuracy in KGQAcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Debate on Graph: a Flexible and Reliable Reasoning Framework for Large Language Models
  2. 2023KnowledgeNavigator: Leveraging Large Language Models for Enhanced Reasoning over Knowledge Graph
  3. 2025Follow the Path: Reasoning over Knowledge Graph Paths to Improve Large Language Model Factuality
  4. 2025ARK-V1: An LLM-Agent for Knowledge Graph Question Answering Requiring Commonsense Reasoning
  5. 2025RJE: A Retrieval-Judgment-Exploration Framework for Efficient Knowledge Graph Question Answering with LLMs
  6. 2024Logic Query of Thoughts: Guiding Large Language Models to Answer Complex Logic Queries with Knowledge Graphs
  7. 2024Decoding on Graphs: Faithful and Sound Reasoning on Knowledge Graphs through Generation of Well-Formed Chains
  8. 2026MARS: Multi-hop Adaptive Retrieval and SPARQL Generation for KGQA