mlsanityData to 14 Oct 2026

LLMs for Knowledge Graph Construction and Reasoning: Recent Capabilities and Future Opportunities

arXiv 2305.13168 ↗First posted 22 May 2023cs.CL cs.AI cs.DBcited by 97

On the map: LLM reasoning → Knowledge- and tool-augmented reasoning

Abstract (arXiv)

This paper presents an exhaustive quantitative and qualitative evaluation of Large Language Models (LLMs) for Knowledge Graph (KG) construction and reasoning. We engage in experiments across eight diverse datasets, focusing on four representative tasks encompassing entity and relation extraction, event extraction, link prediction, and question-answering, thereby thoroughly exploring LLMs' performance in the domain of construction and inference. Empirically, our findings suggest that LLMs, represented by GPT-4, are more suited as inference assistants rather than few-shot information extractors. Specifically, while GPT-4 exhibits good performance in tasks related to KG construction, it excels further in reasoning tasks, surpassing fine-tuned models in certain cases. Moreover, our investigation extends to the potential generalization ability of LLMs for information extraction, leading to the proposition of a Virtual Knowledge Extraction task and the development of the corresponding VINE dataset. Based on these empirical findings, we further propose AutoKG, a multi-agent-based approach employing LLMs and external sources for KG construction and reasoning. We anticipate that this research can provide invaluable insights for future undertakings in the field of knowledge graphs. The code and datasets are in https://github.com/zjunlp/AutoKG.

Key facts. Cited by 97 papers in the map (data through 14 Oct 2026). Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning. Builds on 62 of its 98 references that are in the map. First posted on arXiv 22 May 2023.

Builds on

62 of 98 references.

References · 62

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2017Deep reinforcement learning from human preferencescited by 3,912
  5. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  6. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  7. 2023A Survey of Large Language Modelscited by 2,576
  8. 2022Emergent Abilities of Large Language Modelscited by 2,512
  9. 2019Language Models as Knowledge Bases?cited by 1,936
  10. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,395
  11. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 894
  12. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
38 more
  1. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 638
  2. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 488
  3. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 411
  4. 2023Larger language models do in-context learning differentlycited by 394
  5. 2018Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Constructioncited by 322
  6. 2021Symbolic Knowledge Distillation: from General Language Models to Commonsense Modelscited by 270
  7. 2020COMET-ATOMIC 2020: On Symbolic and Neural Commonsense Knowledge Graphscited by 254
  8. 2022Reasoning with Language Model Prompting: A Surveycited by 252
  9. 2023ChatIE: Zero-Shot Information Extraction via Chatting with ChatGPTcited by 250
  10. 2016Key-Value Memory Networks for Directly Reading Documentscited by 223
  11. 2023ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learningcited by 206
  12. 2023Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4cited by 202
  13. 2015Named Entity Recognition with Bidirectional LSTM-CNNscited by 201
  14. 2021KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extractioncited by 180
  15. 2021Retrieving and Reading: A Comprehensive Survey on Open-domain Question Answeringcited by 178
  16. 2017Variational Reasoning for Question Answering with Knowledge Graphcited by 170
  17. 2022A Review on Language Models as Knowledge Basescited by 157
  18. 2023Evaluating ChatGPT's Information Extraction Capabilities: An Assessment of Performance, Explainability, Calibration, and Faithfulnesscited by 130
  19. 2023Large Language Model Is Not a Good Few-shot Information Extractor, but a Good Reranker for Hard Samples!cited by 119
  20. 2023A comprehensive evaluation of ChatGPT's zero-shot Text-to-SQL capabilitycited by 119
  21. 2022A Survey of Knowledge Graph Reasoning on Graph Types: Static, Dynamic, and Multimodalcited by 106
  22. 2020MAVEN: A Massive General Domain Event Detection Datasetcited by 104
  23. 2022ChatGPT Makes Medicine Easy to Swallow: An Exploratory Case Study on Simplified Radiology Reportscited by 101
  24. 2022SimKGC: Simple Contrastive Knowledge Graph Completion with Pre-trained Language Modelscited by 91
  25. 2020Knowledge Graph Embedding for Link Prediction: A Comparative Analysiscited by 85
  26. 2022DecAF: Joint Decoding of Answers and Logical Forms for Question Answering over Knowledge Basescited by 84
  27. 2023Exploring the Feasibility of ChatGPT for Event Extractioncited by 76
  28. 2021Re-TACRED: Addressing Shortcomings of the TACRED Datasetcited by 74
  29. 2022Multi-Modal Knowledge Graph Construction and Application: A Surveycited by 74
  30. 2023Can ChatGPT Replace Traditional KBQA Models? An In-depth Analysis of the Question Answering Performance of the GPT LLM Familycited by 71
  31. 2023Evaluating GPT-4 and ChatGPT on Japanese Medical Licensing Examinationscited by 63
  32. 2023GPT-RE: In-context Learning for Relation Extraction using Large Language Modelscited by 62
  33. 2022Temporal Knowledge Graph Completion: A Surveycited by 61
  34. 2019Meta-Learning with Dynamic-Memory-Based Prototypical Network for Few-Shot Event Detectioncited by 54
  35. 2023ChatGPT: A Meta-Analysis after 2.5 Monthscited by 51
  36. 2021Packed Levitated Marker for Entity and Relation Extractioncited by 49
  37. 2022Generative Knowledge Graph Construction: A Reviewcited by 49
  38. 2023Translating Radiology Reports into Plain Language using ChatGPT and GPT-4 with Prompt Learning: Promising Results, Limitations, and Potentialcited by 39

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 14; 2024: 27; 2025: 32; 2026: 24.

97 papers.

Sort:

By area:

  1. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 668
  2. 2023Exploring Collaboration Mechanisms for LLM Agents: A Social Psychology Viewcited by 222
  3. 2023When Large Language Models Meet Personalization: Perspectives of Challenges and Opportunitiescited by 90
  4. 2023Cognitive Mirage: A Review of Hallucinations in Large Language Modelscited by 84
  5. 2023Large Language Models and Knowledge Graphs: Opportunities and Challengescited by 71
  6. 2024Demystifying Chains, Trees, and Graphs of Thoughtscited by 63
  7. 2024KG-RAG: Bridging the Gap Between Knowledge and Creativitycited by 56
  8. 2023Making Large Language Models Perform Better in Knowledge Graph Completioncited by 48
  9. 2023Exploring Large Language Models for Knowledge Graph Completioncited by 43
  10. 2024GraphEval: A Knowledge-Graph Based LLM Hallucination Evaluation Frameworkcited by 30
  11. 2025KARMA: Leveraging Multi-Agent LLMs for Automated Knowledge Graph Enrichmentcited by 27
  12. 2025LLM-empowered knowledge graph construction: A surveycited by 21
38 more
  1. 2024A Survey of Large Language Models on Generative Graph Analytics: Query, Learning, and Applicationscited by 17
  2. 2024MKGL: Mastery of a Three-Word Languagecited by 17
  3. 2024Research Trends for the Interplay between Large Language Models and Knowledge Graphscited by 15
  4. 2024Actionable Cyber Threat Intelligence using Knowledge Graphs and Large Language Modelscited by 14
  5. 2024LLM as Prompter: Low-resource Inductive Reasoning on Arbitrary Knowledge Graphscited by 13
  6. 2025AssertionForge: Enhancing Formal Verification Assertion Generation with Structured Representation of Specifications and RTLcited by 13
  7. 2023Large Knowledge Model: Perspectives and Challengescited by 12
  8. 2023Benchmarking the Abilities of Large Language Models for RDF Knowledge Graph Creation and Comprehension: How Well Do LLMs Speak Turtle?cited by 11
  9. 2024Enhancing Supply Chain Visibility with Knowledge Graphs and Large Language Modelscited by 11
  10. 2024iText2KG: Incremental Knowledge Graphs Construction Using Large Language Modelscited by 11
  11. 2024Improving Complex Reasoning over Knowledge Graph with Logic-Aware Curriculum Tuningcited by 10
  12. 2024LLM-Powered Explanations: Unraveling Recommendations Through Subgraph Reasoningcited by 10
  13. 2024No Free Lunch: Retrieval-Augmented Generation Undermines Fairness in LLMs, Even for Vigilant Userscited by 9
  14. 2025Graph-Augmented Reasoning: Evolving Step-by-Step Knowledge Graph Retrieval for LLM Reasoningcited by 8
  15. 2025A Survey on Retrieval And Structuring Augmented Generation with Large Language Modelscited by 8
  16. 2024GenRES: Rethinking Evaluation for Generative Relation Extraction in the Era of Large Language Modelscited by 7
  17. 2023Glitter or Gold? Deriving Structured Insights from Sustainability Reports via Large Language Modelscited by 6
  18. 2023Beyond Isolation: Multi-Agent Synergy for Improving Knowledge Graph Constructioncited by 6
  19. 2024Consolidating Trees of Robotic Plans Generated Using Large Language Models to Improve Reliabilitycited by 6
  20. 2024Large language models as oracles for instantiating ontologies with domain-specific knowledgecited by 6
  21. 2024ProveRAG: Provenance-Driven Vulnerability Analysis with Automated Retrieval-Augmented LLMscited by 6
  22. 2024Neurosymbolic Graph Enrichment for Grounded World Modelscited by 6
  23. 2025How to Mitigate Information Loss in Knowledge Graphs for GraphRAG: Leveraging Triple Context Restoration and Query-Driven Feedbackcited by 6
  24. 2025KG-LLM-Bench: A Scalable Benchmark for Evaluating LLM Reasoning on Textualized Knowledge Graphscited by 6
  25. 2024Uncertainty Management in the Construction of Knowledge Graphs: a Surveycited by 5
  26. 2024Finetuning Generative Large Language Models with Discrimination Instructions for Knowledge Graph Completioncited by 5
  27. 2025Zero-Shot End-to-End Relation Extraction in Chinese: A Comparative Study of Gemini, LLaMA and ChatGPTcited by 5
  28. 2025MedKGent: A Large Language Model Agent Framework for Constructing Temporally Evolving Medical Knowledge Graphcited by 5
  29. 2025Actions Speak Louder than Prompts: A Large-Scale Study of LLMs for Graph Inferencecited by 5
  30. 2023MoCoSA: Momentum Contrast for Knowledge Graph Completion with Structure-Augmented Pre-trained Language Modelscited by 4
  31. 2025MuCo-KGC: Multi-Context-Aware Knowledge Graph Completioncited by 4
  32. 2025LKD-KGC: Domain-Specific KG Construction via LLM-driven Knowledge Dependency Parsingcited by 4
  33. 2025Exploring the Challenges and Opportunities of AI-assisted Codebase Generationcited by 4
  34. 2026KNIGHT: Knowledge Graph-Driven Multiple-Choice Question Generation with Adaptive Hardness Calibrationcited by 4
  35. 2026Coverage-Aware Web Crawling for Domain-Specific Supplier Discovery via a Web--Knowledge--Web Pipelinecited by 4
  36. 2024Assessing LLMs Suitability for Knowledge Graph Completioncited by 3
  37. 2024Retrieval, Reasoning, Re-ranking: A Context-Enriched Framework for Knowledge Graph Completioncited by 3
  38. 2025LLM-KG-Bench 3.0: A Compass for SemanticTechnology Capabilities in the Ocean of LLMscited by 3

50 most cited of 97 All 97 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Thinking with Knowledge Graphs: Enhancing LLM Reasoning Through Structured Data
  2. 2025Grounding LLM Reasoning with Knowledge Graphs
  3. 2023KG-GPT: A General Framework for Reasoning on Knowledge Graphs Using Large Language Models
  4. 2025Reliable Reasoning Path: Distilling Effective Guidance for LLM Reasoning with Knowledge Graphs
  5. 2023Enhancing Knowledge Graph Construction Using Large Language Models
  6. 2023Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graph
  7. 2026LLMs+Graphs: Toward Graph-Native, Synergistic AI Systems
  8. 2024Paths-over-Graph: Knowledge Graph Empowered Large Language Model Reasoning