# Enhancing Knowledge Graph Construction: Evaluating with Emphasis on Hallucination, Omission, and Graph Similarity Metrics

arXiv: https://arxiv.org/abs/2502.05239 · HTML: https://mlsanity.com/paper/2502.05239/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 38 references that are in the map.
- First posted on arXiv 7 Feb 2025.

## Builds on (most cited first, 28 of 28)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [Knowledge Graphs](https://mlsanity.com/paper/2003.02320/) (2020)
- [Zephyr: Direct Distillation of LM Alignment](https://mlsanity.com/paper/2310.16944/) (2023)
- [ChatIE: Zero-Shot Information Extraction via Chatting with ChatGPT](https://mlsanity.com/paper/2302.10205/) (2023)
- [Evaluating ChatGPT's Information Extraction Capabilities: An Assessment of Performance, Explainability, Calibration, and Faithfulness](https://mlsanity.com/paper/2304.11633/) (2023)
- [LLMs for Knowledge Graph Construction and Reasoning: Recent Capabilities and Future Opportunities](https://mlsanity.com/paper/2305.13168/) (2023)
- [ExplaGraphs: An Explanation Graph Generation Task for Structured Commonsense Reasoning](https://mlsanity.com/paper/2104.07644/) (2021)
- [Neural data-to-text generation: A comparison between pipeline and end-to-end architectures](https://mlsanity.com/paper/1908.09022/) (2019)
- [Structured prompt interrogation and recursive extraction of semantics (SPIRES): A method for populating knowledge bases using zero-shot learning](https://mlsanity.com/paper/2304.02711/) (2023)
- [Text2KGBench: A Benchmark for Ontology-Driven Knowledge Graph Generation from Text](https://mlsanity.com/paper/2308.02357/) (2023)
- [Enhancing Knowledge Graph Construction Using Large Language Models](https://mlsanity.com/paper/2305.04676/) (2023)
- [PiVe: Prompting with Iterative Verification Improving Graph-based Generative Capability of LLMs](https://mlsanity.com/paper/2305.12392/) (2023)
- [Exploring Large Language Models for Knowledge Graph Completion](https://mlsanity.com/paper/2308.13916/) (2023)
- [CodeKGC: Code Language Model for Generative Knowledge Graph Construction](https://mlsanity.com/paper/2304.09048/) (2023)
- [Iterative Zero-Shot LLM Prompting for Knowledge Graph Construction](https://mlsanity.com/paper/2307.01128/) (2023)
- [Knowledge Graph Completion Models are Few-shot Learners: An Empirical Study of Relation Labeling in E-commerce with LLMs](https://mlsanity.com/paper/2305.09858/) (2023)
- [Exploring In-Context Learning Capabilities of Foundation Models for Generating Knowledge Graphs from Text](https://mlsanity.com/paper/2305.08804/) (2023)
- [BERT Based Clinical Knowledge Extraction for Biomedical Knowledge Graph Construction and Analysis](https://mlsanity.com/paper/2304.10996/) (2023)
- [Construction and Applications of Billion-Scale Pre-Trained Multimodal Business Knowledge Graph](https://mlsanity.com/paper/2209.15214/) (2022)
- [Relevant Entity Selection: Knowledge Graph Bootstrapping via Zero-Shot Analogical Pruning](https://mlsanity.com/paper/2306.16296/) (2023)
- [A Case Study for Compliance as Code with Graphs and Language Models: Public release of the Regulatory Knowledge Graph](https://mlsanity.com/paper/2302.01842/) (2023)

## Built on by (most cited first, 1 of 1)

- [ATOM: AdapTive and OptiMized dynamic temporal knowledge graph construction using LLMs](https://mlsanity.com/paper/2510.22590/) (2025)

## Abstract (arXiv)

Recent advancements in large language models have demonstrated significant potential in the automated construction of knowledge graphs from unstructured text. This paper builds upon our previous work [16], which evaluated various models using metrics like precision, recall, F1 score, triple matching, and graph matching, and introduces a refined approach to address the critical issues of hallucination and omission. We propose an enhanced evaluation framework incorporating BERTScore for graph similarity, setting a practical threshold of 95% for graph matching. Our experiments focus on the Mistral model, comparing its original and fine-tuned versions in zero-shot and few-shot settings. We further extend our experiments using examples from the KELM-sub training dataset, illustrating that the fine-tuned model significantly improves knowledge graph construction accuracy while reducing the exact hallucination and omission. However, our findings also reveal that the fine-tuned models perform worse in generalization tasks on the KELM-sub dataset. This study underscores the importance of comprehensive evaluation metrics in advancing the state-of-the-art in knowledge graph construction from textual data.