# From human experts to machines: An LLM supported approach to ontology and knowledge graph construction

arXiv: https://arxiv.org/abs/2403.08345 · HTML: https://mlsanity.com/paper/2403.08345/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 26 references that are in the map.
- First posted on arXiv 13 Mar 2024.

## Builds on (most cited first, 15 of 15)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021)
- [Knowledge Graphs](https://mlsanity.com/paper/2003.02320/) (2020)
- [KG-BERT: BERT for Knowledge Graph Completion](https://mlsanity.com/paper/1909.03193/) (2019)
- [Crawling the Internal Knowledge-Base of Language Models](https://mlsanity.com/paper/2301.12810/) (2023)
- [LLMs4OL: Large Language Models for Ontology Learning](https://mlsanity.com/paper/2307.16648/) (2023)
- [Large Language Models and Knowledge Graphs: Opportunities and Challenges](https://mlsanity.com/paper/2308.06374/) (2023)
- [LLM-assisted Knowledge Graph Engineering: Experiments with ChatGPT](https://mlsanity.com/paper/2307.06917/) (2023)
- [Towards Ontology Construction with Language Models](https://mlsanity.com/paper/2309.09898/) (2023)
- [Evaluating Language Models for Knowledge Base Completion](https://mlsanity.com/paper/2303.11082/) (2023)
- [Dynamic Retrieval Augmented Generation of Ontologies using Artificial Intelligence (DRAGON-AI)](https://mlsanity.com/paper/2312.10904/) (2023)
- [Detection and Annotation of Plant Organs from Digitized Herbarium Scans using Deep Learning](https://mlsanity.com/paper/2007.13106/) (2020)
- [Can Large Language Models Augment a Biomedical Ontology with missing Concepts and Relations?](https://mlsanity.com/paper/2311.06858/) (2023)

## Built on by (most cited first, 30 of 40)

- [KGGen: Extracting Knowledge Graphs from Plain Text with Language Models](https://mlsanity.com/paper/2502.09956/) (2025)
- [Combining Knowledge Graphs and Large Language Models](https://mlsanity.com/paper/2407.06564/) (2024)
- [LLMs4Life: Large Language Models for Ontology Learning in Life Sciences](https://mlsanity.com/paper/2412.02035/) (2024)
- [iText2KG: Incremental Knowledge Graphs Construction Using Large Language Models](https://mlsanity.com/paper/2409.03284/) (2024)
- [Towards Next-Generation Urban Decision Support Systems through AI-Powered Construction of Scientific Ontology using Large Language Models -- A Case in Optimizing Intermodal Freight Transportation](https://mlsanity.com/paper/2405.19255/) (2024)
- [Large Language Models for Scholarly Ontology Generation: An Extensive Analysis in the Engineering Field](https://mlsanity.com/paper/2412.08258/) (2024)
- [Leveraging Generative AI for Urban Digital Twins: A Scoping Review on the Autonomous Generation of Urban Data, Scenarios, Designs, and 3D City Models for Smart City Advancement](https://mlsanity.com/paper/2405.19464/) (2024)
- [Multi-Document Financial Question Answering using LLMs](https://mlsanity.com/paper/2411.07264/) (2024)
- [Leveraging LLM for Automated Ontology Extraction and Knowledge Graph Generation](https://mlsanity.com/paper/2412.00608/) (2024)
- [MedKGent: A Large Language Model Agent Framework for Constructing Temporally Evolving Medical Knowledge Graph](https://mlsanity.com/paper/2508.12393/) (2025)
- [Retrieval-Augmented Generation of Ontologies from Relational Databases](https://mlsanity.com/paper/2506.01232/) (2025)
- [LLM-Assisted Knowledge Graph Completion for Curriculum and Domain Modelling in Personalized Higher Education Recommendations](https://mlsanity.com/paper/2501.12300/) (2025)
- [OntoRAG: Enhancing Question-Answering through Automated Ontology Derivation from Unstructured Knowledge Bases](https://mlsanity.com/paper/2506.00664/) (2025)
- [Ontology-to-tools compilation for executable semantic constraint enforcement in LLM agents](https://mlsanity.com/paper/2602.03439/) (2026)
- [Automating Intervention Discovery from Scientific Literature: A Progressive Ontology Prompting and Dual-LLM Framework](https://mlsanity.com/paper/2409.00054/) (2024)
- [Guiding Generative Storytelling with Knowledge Graphs](https://mlsanity.com/paper/2505.24803/) (2025)
- [Bridging Dual Knowledge Graphs for Multi-Hop Question Answering in Construction Safety](https://mlsanity.com/paper/2507.13625/) (2025)
- [LINK-KG: LLM-Driven Coreference-Resolved Knowledge Graphs for Human Smuggling Networks](https://mlsanity.com/paper/2510.26486/) (2025)
- [Inside CORE-KG: Evaluating Structured Prompting and Coreference Resolution for Knowledge Graphs](https://mlsanity.com/paper/2510.26512/) (2025)
- [LLM-Driven Ontology Construction for Enterprise Knowledge Graphs](https://mlsanity.com/paper/2602.01276/) (2026)
- [Knowledge Pyramid Construction for Multi-Level Retrieval-Augmented Generation](https://mlsanity.com/paper/2407.21276/) (2024)
- [DSTI at LLMs4OL 2024 Task A: Intrinsic versus extrinsic knowledge for type classification](https://mlsanity.com/paper/2408.14236/) (2024)
- [Harnessing multiple LLMs for Information Retrieval: A case study on Deep Learning methodologies in Biodiversity publications](https://mlsanity.com/paper/2411.09269/) (2024)
- [KG2QA: Knowledge Graph-enhanced Retrieval-augmented Generation for Communication Standards Question Answering](https://mlsanity.com/paper/2506.07037/) (2025)
- [Integrating Graphs, Large Language Models, and Agents: Reasoning and Retrieval](https://mlsanity.com/paper/2604.15951/) (2026)
- [Towards Automated Ontology Generation from Unstructured Text: A Multi-Agent LLM Approach](https://mlsanity.com/paper/2604.23090/) (2026)
- [The Mercurial Top-Level Ontology of Large Language Models](https://mlsanity.com/paper/2405.01581/) (2024)
- [Affordance Perception by a Knowledge-Guided Vision-Language Model with Efficient Error Correction](https://mlsanity.com/paper/2407.13368/) (2024)
- [In-Context Learning with Topological Information for Knowledge Graph Completion](https://mlsanity.com/paper/2412.08742/) (2024)
- [LLM-based Zero-shot Triple Extraction for Automated Ontology Generation from Software Engineering Standards](https://mlsanity.com/paper/2509.00140/) (2025)

## Abstract (arXiv)

The conventional process of building Ontologies and Knowledge Graphs (KGs) heavily relies on human domain experts to define entities and relationship types, establish hierarchies, maintain relevance to the domain, fill the ABox (or populate with instances), and ensure data quality (including amongst others accuracy and completeness). On the other hand, Large Language Models (LLMs) have recently gained popularity for their ability to understand and generate human-like natural language, offering promising ways to automate aspects of this process. This work explores the (semi-)automatic construction of KGs facilitated by open-source LLMs. Our pipeline involves formulating competency questions (CQs), developing an ontology (TBox) based on these CQs, constructing KGs using the developed ontology, and evaluating the resultant KG with minimal to no involvement of human experts. We showcase the feasibility of our semi-automated pipeline by creating a KG on deep learning methodologies by exploiting scholarly publications. To evaluate the answers generated via Retrieval-Augmented-Generation (RAG) as well as the KG concepts automatically extracted using LLMs, we design a judge LLM, which rates the generated content based on ground truth. Our findings suggest that employing LLMs could potentially reduce the human effort involved in the construction of KGs, although a human-in-the-loop approach is recommended to evaluate automatically generated KGs.