mlsanityData to 7 Oct 2026

Generate-on-Graph: Treat LLM as both Agent and KG in Incomplete Knowledge Graph Question Answering

arXiv 2404.14741 ↗First posted 23 Apr 2024cs.CL cs.AIcited by 59

On the map: LLM agents and tool use → Web, search, and research agents

Abstract (arXiv)

To address the issues of insufficient knowledge and hallucination in Large Language Models (LLMs), numerous studies have explored integrating LLMs with Knowledge Graphs (KGs). However, these methods are typically evaluated on conventional Knowledge Graph Question Answering (KGQA) with complete KGs, where all factual triples required for each question are entirely covered by the given KG. In such cases, LLMs primarily act as an agent to find answer entities within the KG, rather than effectively integrating the internal knowledge of LLMs and external knowledge sources such as KGs. In fact, KGs are often incomplete to cover all the knowledge required to answer questions. To simulate these real-world scenarios and evaluate the ability of LLMs to integrate internal and external knowledge, we propose leveraging LLMs for QA under Incomplete Knowledge Graph (IKGQA), where the provided KG lacks some of the factual triples for each question, and construct corresponding datasets. To handle IKGQA, we propose a training-free method called Generate-on-Graph (GoG), which can generate new factual triples while exploring KGs. Specifically, GoG performs reasoning through a Thinking-Searching-Generating framework, which treats LLM as both Agent and KG in IKGQA. Experimental results on two datasets demonstrate that our GoG outperforms all previous methods.

Key facts. Cited by 59 papers in the map (data through 7 Oct 2026). Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. Builds on 24 of its 45 references that are in the map. First posted on arXiv 23 Apr 2024.

Builds on

24 of 45 references.

References · 24

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  5. 2023Qwen Technical Reportcited by 3,273
  6. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  7. 2022A Survey on In-context Learningcited by 717
  8. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 663
  9. 2020A Survey on Knowledge Graphs: Representation, Acquisition and Applicationscited by 643
  10. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 554
  11. 2018The Web as a Knowledge-base for Answering Complex Questionscited by 409
  12. 2022Complexity-Based Prompting for Multi-Step Reasoningcited by 387
12 more
  1. 2023StructGPT: A General Framework for Large Language Model to Reason over Structured Datacited by 339
  2. 2023Reasoning on Graphs: Faithful and Interpretable Large Language Model Reasoningcited by 326
  3. 2022Generate rather than Retrieve: Large Language Models are Strong Context Generatorscited by 296
  4. 2023Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graphcited by 235
  5. 2019PullNet: Open Domain Question Answering with Iterative Retrieval on Knowledge Bases and Textcited by 182
  6. 2023Chain-of-Knowledge: Grounding Large Language Models via Dynamic Knowledge Adapting over Heterogeneous Sourcescited by 121
  7. 2023Few-shot In-context Learning for Knowledge Base Question Answeringcited by 74
  8. 2020SPARQA: Skeleton-based Semantic Parsing for Complex Questions over Knowledge Basescited by 47
  9. 2023Graph Reasoning for Question Answering with Triplet Retrievalcited by 34
  10. 2024Call Me When Necessary: LLMs can Efficiently and Faithfully Reason over Structured Environmentscited by 27
  11. 2023keqing: knowledge-based question answering is a nature chain-of-thought mentor of LLMcited by 16
  12. 2023Large Language Models Meet Knowledge Graphs to Answer Factoid Questionscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 34; 2026: 19.

59 papers.

Sort:

By area:

  1. 2024AriGraph: Learning Knowledge Graph World Models with Episodic Memory for LLM Agentscited by 100
  2. 2025Agent KB: Leveraging Cross-Domain Experience for Agentic Problem Solvingcited by 90
  3. 2024FlowBench: Revisiting and Benchmarking Workflow-Guided Planning for LLM-based Agentscited by 39
  4. 2025Rethinking Memory in LLM based Agents: Representations, Operations, and Emerging Topicscited by 36
  5. 2024Knowledge Graphs, Large Language Models, and Hallucinations: An NLP Perspectivecited by 29
  6. 2025GraphMaster: Automated Graph Synthesis via LLM Agents in Data-Limited Environmentscited by 18
  7. 2025Graph2text or Graph2token: A Perspective of Large Language Models for Graph Learningcited by 12
  8. 2024Knowledge Graph Enhanced Language Agents for Recommendationcited by 11
  9. 2025Advancing Language Multi-Agent Learning with Credit Re-Assignment for Interactive Environment Generalizationcited by 11
  10. 2025Diagnosing and Addressing Pitfalls in KG-RAG Datasets: Toward More Reliable Benchmarkingcited by 11
  11. 2025Memory-augmented Query Reconstruction for LLM-based Knowledge Graph Reasoningcited by 10
  12. 2025What Breaks Knowledge Graph based RAG? Benchmarking and Empirical Insights into Reasoning under Incomplete Knowledgecited by 9
38 more
  1. 2026PrivGemo: Privacy-Preserving Dual-Tower Graph Retrieval for Empowering LLM Reasoning with Memory Augmentationcited by 9
  2. 2025EarthSE: A Benchmark for Evaluating Earth Scientific Exploration Capability of LLMscited by 7
  3. 2024Way to Specialist: Closing Loop Between Specialized LLM and Evolving Domain Knowledge Graphcited by 6
  4. 2026Relink: Constructing Query-Driven Evidence Graph On-the-Fly for GraphRAGcited by 6
  5. 2026GraphWalker: Agentic Knowledge Graph Question Answering via Synthetic Trajectory Curriculumcited by 6
  6. 2024LLM-based Discriminative Reasoning for Knowledge Graph Question Answeringcited by 5
  7. 2025Search-on-Graph: Iterative Informed Navigation for Large Language Model Reasoning on Knowledge Graphscited by 5
  8. 2026A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Datacited by 5
  9. 2026RADAR: Redundancy-Aware Diffusion for Multi-Agent Communication Structure Generationcited by 5
  10. 2025HealthGenie: Empowering Users with Healthy Dietary Guidance through Knowledge Graph and Large Language Modelscited by 4
  11. 2026RebuttalAgent: Strategic Persuasion in Academic Rebuttal via Theory of Mindcited by 4
  12. 2025KnowPath: Knowledge-enhanced Reasoning via LLM-generated Inference Paths over Knowledge Graphscited by 3
  13. 2025Incomplete Graph Learning: A Comprehensive Surveycited by 3
  14. 2025Advancing Multi-Agent RAG Systems with Minimalist Reinforcement Learningcited by 3
  15. 2025Knowledge-Level Consistency Reinforcement Learning: Dual-Fact Alignment for Long-Form Factualitycited by 3
  16. 2025Debate over Mixed-knowledge: A Robust Multi-Agent Reasoning Framework for Incomplete Knowledge Graph Question Answeringcited by 3
  17. 2026CoG: Controllable Graph Reasoning via Relational Blueprints and Failure-Aware Refinement over Knowledge Graphscited by 3
  18. 2025R2-KG: General-Purpose Dual-Agent Framework for Reliable Reasoning on Knowledge Graphscited by 2
  19. 2025Beyond Chains: Bridging Large Language Models and Knowledge Bases in Complex Question Answeringcited by 2
  20. 2025MultiHal: Multilingual Dataset for Knowledge-Graph Grounded Evaluation of LLM Hallucinationscited by 2
  21. 2025LLM Inference Enhanced by External Knowledge: A Surveycited by 2
  22. 2025Towards Open-World Retrieval-Augmented Generation on Knowledge Graph: A Multi-Agent Collaboration Frameworkcited by 2
  23. 2025Improving Factuality in LLMs via Inference-Time Knowledge Graph Constructioncited by 2
  24. 2025Hybrid Fact-Checking that Integrates Knowledge Graphs, Large Language Models, and Search-Based Retrieval Agents Improves Interpretable Claim Verificationcited by 2
  25. 2025Self-Correction Distillation for Structured Data Question Answeringcited by 2
  26. 2025GR-Agent: Adaptive Graph Reasoning Agent under Incomplete Knowledgecited by 2
  27. 2026Leveraging LLM-GNN Integration for Open-World Question Answering over Knowledge Graphscited by 2
  28. 2026Code-on-Graph: Iterative Programmatic Reasoning via Large Language Models on Knowledge Graphscited by 2
  29. 2025PASemiQA: Plan-Assisted Agent for Question Answering on Semi-Structured Data with Text and Relational Informationcited by 1
  30. 2025Human Cognition Inspired RAG with Knowledge Graph for Complex Problem Solvingcited by 1
  31. 2025Question-Aware Knowledge Graph Prompting for Enhancing Large Language Modelscited by 1
  32. 2025A Case Study of Cross-Lingual Zero-Shot Generalization for Classical Languages in LLMscited by 1
  33. 2025KG-CQR: Leveraging Structured Relation Representations in Knowledge Graphs for Contextual Query Retrievalcited by 1
  34. 2025The Role of Exploration Modules in Small Language Models for Knowledge Graph Question Answeringcited by 1
  35. 2025Plan Then Retrieve: Reinforcement Learning-Guided Complex Reasoning over Knowledge Graphscited by 1
  36. 2025Encoder-Free Knowledge-Graph Reasoning with LLMs via Hyperdimensional Path Retrievalcited by 1
  37. 2026Failure is Feedback: History-Aware Backtracking for Agentic Traversal in Multimodal Graphscited by 1
  38. 2026GAM-RAG: Gain-Adaptive Memory for Evolving Retrieval in Retrieval-Augmented Generationcited by 1

50 most cited of 59 All 59 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Fusing Knowledge and Language: A Comparative Study of Knowledge Graph-Based Question Answering with LLMs
  2. 2024GS-KGC: A Generative Subgraph-based Framework for Knowledge Graph Completion with Large Language Models
  3. 2023LLMs for Knowledge Graph Construction and Reasoning: Recent Capabilities and Future Opportunities
  4. 2024Logic Query of Thoughts: Guiding Large Language Models to Answer Complex Logic Queries with Knowledge Graphs
  5. 2024Can Knowledge Graphs Make Large Language Models More Trustworthy? An Empirical Study Over Open-ended Question Answering
  6. 2026A Human-in-the-Loop, LLM-Centered Architecture for Knowledge-Graph Question Answering
  7. 2025GraphGen: Enhancing Supervised Fine-Tuning for LLMs with Knowledge-Driven Synthetic Data Generation
  8. 2025Knowledge Graph-extended Retrieval Augmented Generation for Question Answering