# Evaluating Class Membership Relations in Knowledge Graphs using Large Language Models

arXiv: https://arxiv.org/abs/2404.17000 · HTML: https://mlsanity.com/paper/2404.17000/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 39 references that are in the map.
- First posted on arXiv 25 Apr 2024.

## Builds on (most cited first, 24 of 24)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting](https://mlsanity.com/paper/2305.04388/) (2023)
- [Unifying Large Language Models and Knowledge Graphs: A Roadmap](https://mlsanity.com/paper/2306.08302/) (2023)
- [Knowledge Graphs](https://mlsanity.com/paper/2003.02320/) (2020)
- [A Survey on Automated Fact-Checking](https://mlsanity.com/paper/2108.11896/) (2021)
- [Generating Fact Checking Explanations](https://mlsanity.com/paper/2004.05773/) (2020)
- [Faithfulness vs. Plausibility: On the (Un)Reliability of Explanations from Large Language Models](https://mlsanity.com/paper/2402.04614/) (2024)
- [A Study of the Quality of Wikidata](https://mlsanity.com/paper/2107.00156/) (2021)
- [Prompting as Probing: Using Language Models for Knowledge Base Construction](https://mlsanity.com/paper/2208.11057/) (2022)
- [Ontology engineering with Large Language Models](https://mlsanity.com/paper/2307.16699/) (2023)
- [Efficient Knowledge Graph Accuracy Evaluation](https://mlsanity.com/paper/1907.09657/) (2019)
- [Construction of Knowledge Graphs: State and Challenges](https://mlsanity.com/paper/2302.11509/) (2023)
- [Knowledge Graphs for the Life Sciences: Recent Developments, Challenges and Opportunities](https://mlsanity.com/paper/2309.17255/) (2023)
- [Knowledge Engineering using Large Language Models](https://mlsanity.com/paper/2310.00637/) (2023)
- [Information Extraction From Co-Occurring Similar Entities](https://mlsanity.com/paper/2102.05444/) (2021)
- [The CaLiGraph Ontology as a Challenge for OWL Reasoners](https://mlsanity.com/paper/2110.05028/) (2021)
- [ProVe: A Pipeline for Automated Provenance Verification of Knowledge Graphs against Textual Sources](https://mlsanity.com/paper/2210.14846/) (2022)
- [Describing and Organizing Semantic Web and Machine Learning Systems in the SWeMLS-KG](https://mlsanity.com/paper/2303.15113/) (2023)
- [Using Large Language Models for Knowledge Engineering (LLMKE): A Case Study on Wikidata](https://mlsanity.com/paper/2309.08491/) (2023)
- [Completing and Debugging Ontologies: state of the art and challenges](https://mlsanity.com/paper/1908.03171/) (2019)
- [Transformer-based Subject Entity Detection in Wikipedia Listings](https://mlsanity.com/paper/2210.01482/) (2022)

## Abstract (arXiv)

A backbone of knowledge graphs are their class membership relations, which assign entities to a given class. As part of the knowledge engineering process, we propose a new method for evaluating the quality of these relations by processing descriptions of a given entity and class using a zero-shot chain-of-thought classifier that uses a natural language intensional definition of a class. We evaluate the method using two publicly available knowledge graphs, Wikidata and CaLiGraph, and 7 large language models. Using the gpt-4-0125-preview large language model, the method's classification performance achieves a macro-averaged F1-score of 0.830 on data from Wikidata and 0.893 on data from CaLiGraph. Moreover, a manual analysis of the classification errors shows that 40.9% of errors were due to the knowledge graphs, with 16.0% due to missing relations and 24.9% due to incorrectly asserted relations. These results show how large language models can assist knowledge engineers in the process of knowledge graph refinement. The code and data are available on Github.