mlsanityData to 14 Oct 2026

Text2KGBench: A Benchmark for Ontology-Driven Knowledge Graph Generation from Text

arXiv 2308.02357 ↗First posted 4 Aug 2023cs.CL cs.AIcited by 48

Not on the map. Why?

Abstract (arXiv)

The recent advances in large language models (LLM) and foundation models with emergent capabilities have been shown to improve the performance of many NLP tasks. LLMs and Knowledge Graphs (KG) can complement each other such that LLMs can be used for KG construction or completion while existing KGs can be used for different tasks such as making LLM outputs explainable or fact-checking in Neuro-Symbolic manner. In this paper, we present Text2KGBench, a benchmark to evaluate the capabilities of language models to generate KGs from natural language text guided by an ontology. Given an input ontology and a set of sentences, the task is to extract facts from the text while complying with the given ontology (concepts, relations, domain/range constraints) and being faithful to the input sentences. We provide two datasets (i) Wikidata-TekGen with 10 ontologies and 13,474 sentences and (ii) DBpedia-WebNLG with 19 ontologies and 4,860 sentences. We define seven evaluation metrics to measure fact extraction performance, ontology conformance, and hallucinations by LLMs. Furthermore, we provide results for two baseline models, Vicuna-13B and Alpaca-LoRA-13B using automatic prompt generation from test cases. The baseline results show that there is room for improvement using both Semantic Web and Natural Language Processing techniques.

Key facts. Cited by 48 papers in the map (data through 14 Oct 2026). Builds on 36 of its 55 references that are in the map. First posted on arXiv 4 Aug 2023.

Builds on

36 of 55 references.

References · 36

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  8. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  9. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  11. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  12. 2017Deep reinforcement learning from human preferencescited by 3,912
24 more
  1. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  2. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  3. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  4. 2020Learning to summarize from human feedbackcited by 2,588
  5. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  6. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  7. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  8. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  9. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  10. 2020Knowledge Graphscited by 621
  11. 2021Large Dual Encoders Are Generalizable Retrieverscited by 456
  12. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 411
  13. 2018Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Constructioncited by 322
  14. 2018FewRel: A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of-the-Art Evaluationcited by 254
  15. 2019DocRED: A Large-Scale Document-Level Relation Extraction Datasetcited by 239
  16. 2021Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Modelscited by 132
  17. 2023Unlimiformer: Long-Range Transformers with Unlimited Length Inputcited by 128
  18. 2020SciREX: A Challenge Dataset for Document-Level Information Extractioncited by 113
  19. 2019FewRel 2.0: Towards More Challenging Few-Shot Relation Classificationcited by 94
  20. 2022Prompting as Probing: Using Language Models for Knowledge Base Constructioncited by 32
  21. 2023CodeKGC: Code Language Model for Generative Knowledge Graph Constructioncited by 28
  22. 2023Evaluating Language Models for Knowledge Base Completioncited by 20
  23. 2023Exploring In-Context Learning Capabilities of Foundation Models for Generating Knowledge Graphs from Textcited by 19
  24. 2021DiS-ReX: A Multilingual Dataset for Distantly Supervised Relation Extractioncited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 13; 2025: 16; 2026: 15.

48 papers.

Sort:

By area:

  1. 2023Cognitive Mirage: A Review of Hallucinations in Large Language Modelscited by 84
  2. 2023Symbol-LLM: Towards Foundational Symbol-centric Interface For Large Language Modelscited by 39
  3. 2024iText2KG: Incremental Knowledge Graphs Construction Using Large Language Modelscited by 11
  4. 2024Retrieval-Augmented Generation-based Relation Extractioncited by 9
  5. 2023Insights into Classifying and Mitigating LLMs' Hallucinationscited by 6
  6. 2024Scaling Knowledge Graph Construction through Synthetic Data Generation and Distillationcited by 6
  7. 2024Way to Specialist: Closing Loop Between Specialized LLM and Evolving Domain Knowledge Graphcited by 6
  8. 2025Automatic Prompt Optimization for Knowledge Graph Construction: Insights from an Empirical Studycited by 6
  9. 2024KGValidator: A Framework for Automatic Validation of Knowledge Graph Constructioncited by 5
  10. 2023From Large Language Models to Knowledge Graphs for Biomarker Discovery in Cancercited by 4
  11. 2024Ontology Population using LLMscited by 4
  12. 2024Human Evaluation of Procedural Knowledge Graph Extraction from Text with Large Language Modelscited by 4
36 more
  1. 2025Retrieval-Augmented Generation of Ontologies from Relational Databasescited by 4
  2. 2025Ontology-Guided, Hybrid Prompt Learning for Generalization in Knowledge Graph Question Answeringcited by 3
  3. 2025LLM-KG-Bench 3.0: A Compass for SemanticTechnology Capabilities in the Ocean of LLMscited by 3
  4. 2025InfraGPT Smart Infrastructure: An End-to-End VLM-Based Framework for Detecting and Managing Urban Defectscited by 3
  5. 2024Prompt-Time Ontology-Driven Symbolic Knowledge Capture with Large Language Modelscited by 2
  6. 2024Automating Intervention Discovery from Scientific Literature: A Progressive Ontology Prompting and Dual-LLM Frameworkcited by 2
  7. 2024Scholarly Wikidata: Population and Exploration of Conference Data in Wikidata using LLMscited by 2
  8. 2025OntologyRAG: Better and Faster Biomedical Code Mapping with Retrieval-Augmented Generation (RAG) Leveraging Ontology Knowledge Graphs and Large Language Modelscited by 2
  9. 2025Structuring the Unstructured: A Systematic Review of Text-to-Structure Generation for Agentic AI with a Universal Evaluation Frameworkcited by 2
  10. 2026LLM-Driven Ontology Construction for Enterprise Knowledge Graphscited by 2
  11. 2024Knowledge Pyramid Construction for Multi-Level Retrieval-Augmented Generationcited by 1
  12. 2024ASGM-KG: Unveiling Alluvial Gold Mining Through Knowledge Graphscited by 1
  13. 2025Enhancing Knowledge Graph Construction: Evaluating with Emphasis on Hallucination, Omission, and Graph Similarity Metricscited by 1
  14. 2025Exploring a Large Language Model for Transforming Taxonomic Data into OWL: Lessons Learned and Implications for Ontology Developmentcited by 1
  15. 2025How do Scaling Laws Apply to Knowledge Graph Engineering Tasks? The Impact of Model Size on Large Language Model Performancecited by 1
  16. 2026Integrating Graphs, Large Language Models, and Agents: Reasoning and Retrievalcited by 1
  17. 2026Evaluation of Pipelines for Data Integration into Knowledge Graphscited by 1
  18. 2026Better Later Than Sooner: Neuro-Symbolic Knowledge Graph Construction via Ontology-grounded Post-extraction Correctioncited by 1
  19. 2024Science Checker Reloaded: A Bidirectional Paradigm for Transparency and Logical Reasoningcited by 0
  20. 2025EMERGE: A Benchmark for Updating Knowledge Graphs with Emerging Textual Knowledgecited by 0
  21. 2025Systematic Evaluation of Knowledge Graph Repair with Large Language Modelscited by 0
  22. 2025Data Overdose? Time for a Quadruple Shot: Knowledge Graph Construction using Enhanced Triple Extractioncited by 0
  23. 2025TextMineX: Data, Evaluation Framework and Ontology-guided LLM Pipeline for Humanitarian Mine Actioncited by 0
  24. 2025Overcoming the Generalization Limits of SLM Finetuning for Shape-Based Extraction of Datatype and Object Propertiescited by 0
  25. 2025OpenGloss: A Synthetic Encyclopedic Dictionary and Semantic Knowledge Graphcited by 0
  26. 2026LEC-KG: An LLM-Embedding Collaborative Framework for Domain-Specific Knowledge Graph Construction -- A Case Study on SDGscited by 0
  27. 2026LLM-based Triplet Extraction from Financial Reportscited by 0
  28. 2026Beyond Predefined Schemas: TRACE-KG for Context-Enriched Knowledge Graph Generationcited by 0
  29. 2026A Benchmark for Gap and Overlap Analysis as a Test of KG Task Readinesscited by 0
  30. 2026A Unified Benchmark for Evaluating Knowledge Graph Construction Methods and Graph Neural Networkscited by 0
  31. 2026Tacit Knowledge Extraction via Logic Augmented Generation and Active Inferencecited by 0
  32. 2026Reversing Arrows in Large Language Modelscited by 0
  33. 2026A Grounded and Decomposed Framework for Relation-Level Hallucination Evaluation in Abstractive Summarizationcited by 0
  34. 2026Propagating construction-time knowledge quality into medical question answering: A framework grounded in clinical guidelinescited by 0
  35. 2026Seed-Anchored Budget-Bounded Graph Rendering for Question Answering on Industry-Standard Power-Grid Information and Exchange Modelscited by 0
  36. 2026Extracting ontology-compliant knowledge from scientific text describing irradiated materials using large language modelscited by 0