# Large Language Models Assisting Ontology Evaluation

arXiv: https://arxiv.org/abs/2507.14552 · HTML: https://mlsanity.com/paper/2507.14552/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 34 references that are in the map.
- First posted on arXiv 19 Jul 2025.

## Builds on (most cited first, 8 of 8)

- [Ontology Generation using Large Language Models](https://mlsanity.com/paper/2503.05388/) (2025)
- [ArCo: the Italian Cultural Heritage Knowledge Graph](https://mlsanity.com/paper/1905.02840/) (2019)
- [OntoChat: a Framework for Conversational Ontology Engineering using Language Models](https://mlsanity.com/paper/2403.05921/) (2024)
- [An Experiment in Retrofitting Competency Questions for Existing Ontologies](https://mlsanity.com/paper/2311.05662/) (2023)
- [Assessing the Capability of Large Language Models for Domain-Specific Ontology Generation](https://mlsanity.com/paper/2504.17402/) (2025)
- [The Water Health Open Knowledge Graph](https://mlsanity.com/paper/2305.11051/) (2023)
- [Toward a Method to Generate Capability Ontologies from Natural Language Descriptions](https://mlsanity.com/paper/2406.07962/) (2024)
- [My Ontologist: Evaluating BFO-Based AI for Definition Support](https://mlsanity.com/paper/2407.17657/) (2024)

## Built on by (most cited first, 2 of 2)

- [OntoExtend: A Framework for Requirement-driven and Scalable Ontology Extension with LLMs](https://mlsanity.com/paper/2607.17963/) (2026)
- [When CQs Go Wrong: Challenges in CQ Verification with OE-Assist](https://mlsanity.com/paper/2606.24619/) (2026)

## Abstract (arXiv)

Ontology evaluation through functional requirements, such as testing via competency question (CQ) verification, is a well-established yet costly, labour-intensive, and error-prone endeavour, even for ontology engineering experts. In this work, we introduce OE-Assist, a novel framework designed to assist ontology evaluation through automated and semi-automated CQ verification. By presenting and leveraging a dataset of 1,393 CQs paired with corresponding ontologies and ontology stories, our contributions present, to our knowledge, the first systematic investigation into large language model (LLM)-assisted ontology evaluation, and include: (i) evaluating the effectiveness of a LLM-based approach for automatically performing CQ verification against a manually created gold standard, and (ii) developing and assessing an LLM-powered framework to assist CQ verification with Prot'eg'e, by providing suggestions. We found that automated LLM-based evaluation with o1-preview and o3-mini perform at a similar level to the average user's performance.