# Text2KGBench: A Benchmark for Ontology-Driven Knowledge Graph Generation from Text

arXiv: https://arxiv.org/abs/2308.02357 · HTML: https://mlsanity.com/paper/2308.02357/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 55 references that are in the map.
- First posted on arXiv 4 Aug 2023.

## Builds on (most cited first, 30 of 36)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [An Explanation of In-context Learning as Implicit Bayesian Inference](https://mlsanity.com/paper/2111.02080/) (2021)
- [Knowledge Graphs](https://mlsanity.com/paper/2003.02320/) (2020)
- [Large Dual Encoders Are Generalizable Retrievers](https://mlsanity.com/paper/2112.07899/) (2021)
- [Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond](https://mlsanity.com/paper/2304.13712/) (2023)
- [Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Construction](https://mlsanity.com/paper/1808.09602/) (2018)
- [FewRel: A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of-the-Art Evaluation](https://mlsanity.com/paper/1810.10147/) (2018)
- [DocRED: A Large-Scale Document-Level Relation Extraction Dataset](https://mlsanity.com/paper/1906.06127/) (2019)
- [Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models](https://mlsanity.com/paper/2102.00894/) (2021)
- [Unlimiformer: Long-Range Transformers with Unlimited Length Input](https://mlsanity.com/paper/2305.01625/) (2023)
- [SciREX: A Challenge Dataset for Document-Level Information Extraction](https://mlsanity.com/paper/2005.00512/) (2020)

## Built on by (most cited first, 30 of 48)

- [Cognitive Mirage: A Review of Hallucinations in Large Language Models](https://mlsanity.com/paper/2309.06794/) (2023)
- [Symbol-LLM: Towards Foundational Symbol-centric Interface For Large Language Models](https://mlsanity.com/paper/2311.09278/) (2023)
- [iText2KG: Incremental Knowledge Graphs Construction Using Large Language Models](https://mlsanity.com/paper/2409.03284/) (2024)
- [Retrieval-Augmented Generation-based Relation Extraction](https://mlsanity.com/paper/2404.13397/) (2024)
- [Insights into Classifying and Mitigating LLMs' Hallucinations](https://mlsanity.com/paper/2311.08117/) (2023)
- [Scaling Knowledge Graph Construction through Synthetic Data Generation and Distillation](https://mlsanity.com/paper/2410.16597/) (2024)
- [Way to Specialist: Closing Loop Between Specialized LLM and Evolving Domain Knowledge Graph](https://mlsanity.com/paper/2411.19064/) (2024)
- [Automatic Prompt Optimization for Knowledge Graph Construction: Insights from an Empirical Study](https://mlsanity.com/paper/2506.19773/) (2025)
- [KGValidator: A Framework for Automatic Validation of Knowledge Graph Construction](https://mlsanity.com/paper/2404.15923/) (2024)
- [From Large Language Models to Knowledge Graphs for Biomarker Discovery in Cancer](https://mlsanity.com/paper/2310.08365/) (2023)
- [Ontology Population using LLMs](https://mlsanity.com/paper/2411.01612/) (2024)
- [Human Evaluation of Procedural Knowledge Graph Extraction from Text with Large Language Models](https://mlsanity.com/paper/2412.03589/) (2024)
- [Retrieval-Augmented Generation of Ontologies from Relational Databases](https://mlsanity.com/paper/2506.01232/) (2025)
- [Ontology-Guided, Hybrid Prompt Learning for Generalization in Knowledge Graph Question Answering](https://mlsanity.com/paper/2502.03992/) (2025)
- [LLM-KG-Bench 3.0: A Compass for SemanticTechnology Capabilities in the Ocean of LLMs](https://mlsanity.com/paper/2505.13098/) (2025)
- [InfraGPT Smart Infrastructure: An End-to-End VLM-Based Framework for Detecting and Managing Urban Defects](https://mlsanity.com/paper/2510.16017/) (2025)
- [Prompt-Time Ontology-Driven Symbolic Knowledge Capture with Large Language Models](https://mlsanity.com/paper/2405.14012/) (2024)
- [Automating Intervention Discovery from Scientific Literature: A Progressive Ontology Prompting and Dual-LLM Framework](https://mlsanity.com/paper/2409.00054/) (2024)
- [Scholarly Wikidata: Population and Exploration of Conference Data in Wikidata using LLMs](https://mlsanity.com/paper/2411.08696/) (2024)
- [OntologyRAG: Better and Faster Biomedical Code Mapping with Retrieval-Augmented Generation (RAG) Leveraging Ontology Knowledge Graphs and Large Language Models](https://mlsanity.com/paper/2502.18992/) (2025)
- [Structuring the Unstructured: A Systematic Review of Text-to-Structure Generation for Agentic AI with a Universal Evaluation Framework](https://mlsanity.com/paper/2508.12257/) (2025)
- [LLM-Driven Ontology Construction for Enterprise Knowledge Graphs](https://mlsanity.com/paper/2602.01276/) (2026)
- [Knowledge Pyramid Construction for Multi-Level Retrieval-Augmented Generation](https://mlsanity.com/paper/2407.21276/) (2024)
- [ASGM-KG: Unveiling Alluvial Gold Mining Through Knowledge Graphs](https://mlsanity.com/paper/2408.08972/) (2024)
- [Enhancing Knowledge Graph Construction: Evaluating with Emphasis on Hallucination, Omission, and Graph Similarity Metrics](https://mlsanity.com/paper/2502.05239/) (2025)
- [Exploring a Large Language Model for Transforming Taxonomic Data into OWL: Lessons Learned and Implications for Ontology Development](https://mlsanity.com/paper/2504.18651/) (2025)
- [How do Scaling Laws Apply to Knowledge Graph Engineering Tasks? The Impact of Model Size on Large Language Model Performance](https://mlsanity.com/paper/2505.16276/) (2025)
- [Integrating Graphs, Large Language Models, and Agents: Reasoning and Retrieval](https://mlsanity.com/paper/2604.15951/) (2026)
- [Evaluation of Pipelines for Data Integration into Knowledge Graphs](https://mlsanity.com/paper/2605.22304/) (2026)
- [Better Later Than Sooner: Neuro-Symbolic Knowledge Graph Construction via Ontology-grounded Post-extraction Correction](https://mlsanity.com/paper/2605.29168/) (2026)

## Abstract (arXiv)

The recent advances in large language models (LLM) and foundation models with emergent capabilities have been shown to improve the performance of many NLP tasks. LLMs and Knowledge Graphs (KG) can complement each other such that LLMs can be used for KG construction or completion while existing KGs can be used for different tasks such as making LLM outputs explainable or fact-checking in Neuro-Symbolic manner. In this paper, we present Text2KGBench, a benchmark to evaluate the capabilities of language models to generate KGs from natural language text guided by an ontology. Given an input ontology and a set of sentences, the task is to extract facts from the text while complying with the given ontology (concepts, relations, domain/range constraints) and being faithful to the input sentences. We provide two datasets (i) Wikidata-TekGen with 10 ontologies and 13,474 sentences and (ii) DBpedia-WebNLG with 19 ontologies and 4,860 sentences. We define seven evaluation metrics to measure fact extraction performance, ontology conformance, and hallucinations by LLMs. Furthermore, we provide results for two baseline models, Vicuna-13B and Alpaca-LoRA-13B using automatic prompt generation from test cases. The baseline results show that there is room for improvement using both Semantic Web and Natural Language Processing techniques.