# Bench4KE: Benchmarking Automated Competency Question Generation

arXiv: https://arxiv.org/abs/2505.24554 · HTML: https://mlsanity.com/paper/2505.24554/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 58 references that are in the map.
- First posted on arXiv 30 May 2025.

## Builds on (most cited first, 11 of 11)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [ArCo: the Italian Cultural Heritage Knowledge Graph](https://mlsanity.com/paper/1905.02840/) (2019)
- [OntoChat: a Framework for Conversational Ontology Engineering using Language Models](https://mlsanity.com/paper/2403.05921/) (2024)
- [An Experiment in Retrofitting Competency Questions for Existing Ontologies](https://mlsanity.com/paper/2311.05662/) (2023)
- [A RAG Approach for Generating Competency Questions in Ontology Engineering](https://mlsanity.com/paper/2409.08820/) (2024)
- [Assessing the Capability of Large Language Models for Domain-Specific Ontology Generation](https://mlsanity.com/paper/2504.17402/) (2025)
- [The African Wildlife Ontology tutorial ontologies: requirements, design, and content](https://mlsanity.com/paper/1905.09519/) (2019)
- [The Water Health Open Knowledge Graph](https://mlsanity.com/paper/2305.11051/) (2023)
- [Improving Ontology Requirements Engineering with OntoChat and Participatory Prompting](https://mlsanity.com/paper/2408.15256/) (2024)
- [Automatically Drafting Ontologies from Competency Questions with FrODO](https://mlsanity.com/paper/2206.02485/) (2022)

## Built on by (most cited first, 3 of 3)

- [VSPO: Validating Semantic Pitfalls in Ontology via LLM-Based CQ Generation](https://mlsanity.com/paper/2511.07991/) (2025)
- [IDEA2: Expert-in-the-loop competency question elicitation for collaborative ontology engineering](https://mlsanity.com/paper/2604.01344/) (2026)
- [When CQs Go Wrong: Challenges in CQ Verification with OE-Assist](https://mlsanity.com/paper/2606.24619/) (2026)

## Abstract (arXiv)

The availability of Large Language Models (LLMs) presents a unique opportunity to reinvigorate research on Knowledge Engineering (KE) automation. This trend is already evident in recent efforts developing LLM-based methods and tools for the automatic generation of Competency Questions (CQs), natural language questions used by ontology engineers to define the functional requirements of an ontology. However, the evaluation of these tools lacks standardization. This undermines the methodological rigor and hinders the replication and comparison of results. To address this gap, we introduce Bench4KE, an extensible API-based benchmarking system for KE automation. The presented release focuses on evaluating tools that generate CQs automatically. Bench4KE provides a curated gold standard consisting of CQ datasets from 17 real-world ontology engineering projects and uses a suite of similarity metrics to assess the quality of the CQs generated. We present a comparative analysis of 6 recent CQ generation systems, which are based on LLMs, establishing a baseline for future research. Bench4KE is also designed to accommodate additional KE automation tasks, such as SPARQL query generation, ontology testing and drafting. Code and datasets are publicly available under the Apache 2.0 license.