# CQ4OE: A benchmark for assessing LLM-assisted ontology generation from competency questions

arXiv: https://arxiv.org/abs/2609.26029 · HTML: https://mlsanity.com/paper/2609.26029/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 51 references that are in the map.
- First posted on arXiv 22 Sep 2026.

## Builds on (most cited first, 8 of 8)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [LLMs4OL: Large Language Models for Ontology Learning](https://mlsanity.com/paper/2307.16648/) (2023)
- [Ontology Generation using Large Language Models](https://mlsanity.com/paper/2503.05388/) (2025)
- [Dynamic Retrieval Augmented Generation of Ontologies using Artificial Intelligence (DRAGON-AI)](https://mlsanity.com/paper/2312.10904/) (2023)
- [A RAG Approach for Generating Competency Questions in Ontology Engineering](https://mlsanity.com/paper/2409.08820/) (2024)
- [Assessing the Capability of Large Language Models for Domain-Specific Ontology Generation](https://mlsanity.com/paper/2504.17402/) (2025)
- [The African Wildlife Ontology tutorial ontologies: requirements, design, and content](https://mlsanity.com/paper/1905.09519/) (2019)
- [Ontology Learning with LLMs: A Benchmark Study on Axiom Identification](https://mlsanity.com/paper/2512.05594/) (2025)

## Abstract (arXiv)

Ontology generation from Competency Questions (CQs) is a central yet labor-intensive phase of Ontology Engineering. While large language models (LLMs) offer promising automation capabilities, current evaluations remain fragmented. Task formulations are heterogeneous, gold standards often lack fine-grained CQ provenance, metrics conflate lexical overlap with structural and logical adequacy, and reference ontologies are not always explicitly designed around the evaluation CQs. Here, we address these limitations with CQ4OE, a benchmark for the systematic and reproducible evaluation of LLM-based ontology generation from CQs. For each ontology in the benchmark, we build a CQ-driven gold OWL ontology with explicit provenance linking each CQ to the classes, properties, and axioms required to answer it. From this resource, we define two complementary evaluation tasks. CQ2Term supports term-level evaluation of CQ-specific class and property prediction over 99 CQs, and CQ2Onto supports ontology-level evaluation over 118 CQs, including hierarchy, property modeling, and axiom-level structure. We demonstrate CQ4OE with experiments using nine LLMs under zero-shot, iterative, and multi-agent generation strategies, showing that LLMs recover explicit vocabulary terms more reliably than creating ontologies, particularly in property modeling, hierarchy construction, and axiom generation.