# LLMs for Knowledge Graph Construction and Reasoning: Recent Capabilities and Future Opportunities

arXiv: https://arxiv.org/abs/2305.13168 · HTML: https://mlsanity.com/paper/2305.13168/

## Key facts

- Cited by 97 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning.
- Builds on 62 of its 98 references that are in the map.
- First posted on arXiv 22 May 2023.

## Placed in

- LLM reasoning / [Knowledge- and tool-augmented reasoning](https://mlsanity.com/lr/knowledge-and-tool-augmented-reasoning/)

## Builds on (most cited first, 30 of 62)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society](https://mlsanity.com/paper/2303.17760/) (2023)
- [A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://mlsanity.com/paper/2302.04023/) (2023)
- [The Flan Collection: Designing Data and Methods for Effective Instruction Tuning](https://mlsanity.com/paper/2301.13688/) (2023)
- [Capabilities of GPT-4 on Medical Challenge Problems](https://mlsanity.com/paper/2303.13375/) (2023)
- [Is ChatGPT a General-Purpose Natural Language Processing Task Solver?](https://mlsanity.com/paper/2302.06476/) (2023)
- [Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond](https://mlsanity.com/paper/2304.13712/) (2023)
- [Larger language models do in-context learning differently](https://mlsanity.com/paper/2303.03846/) (2023)
- [Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Construction](https://mlsanity.com/paper/1808.09602/) (2018)
- [Symbolic Knowledge Distillation: from General Language Models to Commonsense Models](https://mlsanity.com/paper/2110.07178/) (2021)
- [COMET-ATOMIC 2020: On Symbolic and Neural Commonsense Knowledge Graphs](https://mlsanity.com/paper/2010.05953/) (2020)
- [Reasoning with Language Model Prompting: A Survey](https://mlsanity.com/paper/2212.09597/) (2022)
- [ChatIE: Zero-Shot Information Extraction via Chatting with ChatGPT](https://mlsanity.com/paper/2302.10205/) (2023)
- [Key-Value Memory Networks for Directly Reading Documents](https://mlsanity.com/paper/1606.03126/) (2016)
- [ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning](https://mlsanity.com/paper/2304.05613/) (2023)
- [Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4](https://mlsanity.com/paper/2304.03439/) (2023)
- [Named Entity Recognition with Bidirectional LSTM-CNNs](https://mlsanity.com/paper/1511.08308/) (2015)
- [KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extraction](https://mlsanity.com/paper/2104.07650/) (2021)
- [Retrieving and Reading: A Comprehensive Survey on Open-domain Question Answering](https://mlsanity.com/paper/2101.00774/) (2021)
- [Variational Reasoning for Question Answering with Knowledge Graph](https://mlsanity.com/paper/1709.04071/) (2017)
- [A Review on Language Models as Knowledge Bases](https://mlsanity.com/paper/2204.06031/) (2022)
- [Evaluating ChatGPT's Information Extraction Capabilities: An Assessment of Performance, Explainability, Calibration, and Faithfulness](https://mlsanity.com/paper/2304.11633/) (2023)

## Built on by (most cited first, 30 of 97)

- [Unifying Large Language Models and Knowledge Graphs: A Roadmap](https://mlsanity.com/paper/2306.08302/) (2023)
- [Exploring Collaboration Mechanisms for LLM Agents: A Social Psychology View](https://mlsanity.com/paper/2310.02124/) (2023)
- [When Large Language Models Meet Personalization: Perspectives of Challenges and Opportunities](https://mlsanity.com/paper/2307.16376/) (2023)
- [Cognitive Mirage: A Review of Hallucinations in Large Language Models](https://mlsanity.com/paper/2309.06794/) (2023)
- [Large Language Models and Knowledge Graphs: Opportunities and Challenges](https://mlsanity.com/paper/2308.06374/) (2023)
- [Demystifying Chains, Trees, and Graphs of Thoughts](https://mlsanity.com/paper/2401.14295/) (2024)
- [KG-RAG: Bridging the Gap Between Knowledge and Creativity](https://mlsanity.com/paper/2405.12035/) (2024)
- [Making Large Language Models Perform Better in Knowledge Graph Completion](https://mlsanity.com/paper/2310.06671/) (2023)
- [Exploring Large Language Models for Knowledge Graph Completion](https://mlsanity.com/paper/2308.13916/) (2023)
- [GraphEval: A Knowledge-Graph Based LLM Hallucination Evaluation Framework](https://mlsanity.com/paper/2407.10793/) (2024)
- [KARMA: Leveraging Multi-Agent LLMs for Automated Knowledge Graph Enrichment](https://mlsanity.com/paper/2502.06472/) (2025)
- [LLM-empowered knowledge graph construction: A survey](https://mlsanity.com/paper/2510.20345/) (2025)
- [A Survey of Large Language Models on Generative Graph Analytics: Query, Learning, and Applications](https://mlsanity.com/paper/2404.14809/) (2024)
- [MKGL: Mastery of a Three-Word Language](https://mlsanity.com/paper/2410.07526/) (2024)
- [Research Trends for the Interplay between Large Language Models and Knowledge Graphs](https://mlsanity.com/paper/2406.08223/) (2024)
- [Actionable Cyber Threat Intelligence using Knowledge Graphs and Large Language Models](https://mlsanity.com/paper/2407.02528/) (2024)
- [LLM as Prompter: Low-resource Inductive Reasoning on Arbitrary Knowledge Graphs](https://mlsanity.com/paper/2402.11804/) (2024)
- [AssertionForge: Enhancing Formal Verification Assertion Generation with Structured Representation of Specifications and RTL](https://mlsanity.com/paper/2503.19174/) (2025)
- [Large Knowledge Model: Perspectives and Challenges](https://mlsanity.com/paper/2312.02706/) (2023)
- [Benchmarking the Abilities of Large Language Models for RDF Knowledge Graph Creation and Comprehension: How Well Do LLMs Speak Turtle?](https://mlsanity.com/paper/2309.17122/) (2023)
- [Enhancing Supply Chain Visibility with Knowledge Graphs and Large Language Models](https://mlsanity.com/paper/2408.07705/) (2024)
- [iText2KG: Incremental Knowledge Graphs Construction Using Large Language Models](https://mlsanity.com/paper/2409.03284/) (2024)
- [Improving Complex Reasoning over Knowledge Graph with Logic-Aware Curriculum Tuning](https://mlsanity.com/paper/2405.01649/) (2024)
- [LLM-Powered Explanations: Unraveling Recommendations Through Subgraph Reasoning](https://mlsanity.com/paper/2406.15859/) (2024)
- [No Free Lunch: Retrieval-Augmented Generation Undermines Fairness in LLMs, Even for Vigilant Users](https://mlsanity.com/paper/2410.07589/) (2024)
- [Graph-Augmented Reasoning: Evolving Step-by-Step Knowledge Graph Retrieval for LLM Reasoning](https://mlsanity.com/paper/2503.01642/) (2025)
- [A Survey on Retrieval And Structuring Augmented Generation with Large Language Models](https://mlsanity.com/paper/2509.10697/) (2025)
- [GenRES: Rethinking Evaluation for Generative Relation Extraction in the Era of Large Language Models](https://mlsanity.com/paper/2402.10744/) (2024)
- [Glitter or Gold? Deriving Structured Insights from Sustainability Reports via Large Language Models](https://mlsanity.com/paper/2310.05628/) (2023)
- [Beyond Isolation: Multi-Agent Synergy for Improving Knowledge Graph Construction](https://mlsanity.com/paper/2312.03022/) (2023)

## Abstract (arXiv)

This paper presents an exhaustive quantitative and qualitative evaluation of Large Language Models (LLMs) for Knowledge Graph (KG) construction and reasoning. We engage in experiments across eight diverse datasets, focusing on four representative tasks encompassing entity and relation extraction, event extraction, link prediction, and question-answering, thereby thoroughly exploring LLMs' performance in the domain of construction and inference. Empirically, our findings suggest that LLMs, represented by GPT-4, are more suited as inference assistants rather than few-shot information extractors. Specifically, while GPT-4 exhibits good performance in tasks related to KG construction, it excels further in reasoning tasks, surpassing fine-tuned models in certain cases. Moreover, our investigation extends to the potential generalization ability of LLMs for information extraction, leading to the proposition of a Virtual Knowledge Extraction task and the development of the corresponding VINE dataset. Based on these empirical findings, we further propose AutoKG, a multi-agent-based approach employing LLMs and external sources for KG construction and reasoning. We anticipate that this research can provide invaluable insights for future undertakings in the field of knowledge graphs. The code and datasets are in https://github.com/zjunlp/AutoKG.