# A RAG Approach for Generating Competency Questions in Ontology Engineering

arXiv: https://arxiv.org/abs/2409.08820 · HTML: https://mlsanity.com/paper/2409.08820/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge-graph and structured-knowledge RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 7 of its 19 references that are in the map.
- First posted on arXiv 13 Sep 2024.

## Placed in

- Retrieval-augmented generation / [Knowledge-graph and structured-knowledge RAG](https://mlsanity.com/rag/knowledge-graph-and-structured-knowledge-rag/)

## Builds on (most cited first, 7 of 7)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Seven Failure Points When Engineering a Retrieval Augmented Generation System](https://mlsanity.com/paper/2401.05856/) (2024)
- [A Good Prompt Is Worth Millions of Parameters: Low-resource Prompt-based Learning for Vision-Language Models](https://mlsanity.com/paper/2110.08484/) (2021)
- [An Experiment in Retrofitting Competency Questions for Existing Ontologies](https://mlsanity.com/paper/2311.05662/) (2023)
- [BigCQ: A large-scale synthetic dataset of competency question patterns formalized into SPARQL-OWL query templates](https://mlsanity.com/paper/2105.09574/) (2021)
- [Divide and Conquer the EmpiRE: A Community-Maintainable Knowledge Graph of Empirical Research in Requirements Engineering](https://mlsanity.com/paper/2306.16791/) (2023)

## Built on by (most cited first, 7 of 7)

- [Bench4KE: Benchmarking Automated Competency Question Generation](https://mlsanity.com/paper/2505.24554/) (2025)
- [Instructional Goal-Aligned Question Generation for Student Evaluation in Virtual Lab Settings: How Closely Do LLMs Actually Align?](https://mlsanity.com/paper/2510.06411/) (2025)
- [Q2Forge: Minting Competency Questions and SPARQL Queries for Question-Answering Over Knowledge Graphs](https://mlsanity.com/paper/2505.13572/) (2025)
- [VSPO: Validating Semantic Pitfalls in Ontology via LLM-Based CQ Generation](https://mlsanity.com/paper/2511.07991/) (2025)
- [IDEA2: Expert-in-the-loop competency question elicitation for collaborative ontology engineering](https://mlsanity.com/paper/2604.01344/) (2026)
- [A Task-Centric Ontology and Deterministic Domain Rules as a Verifiable Core for AI-Assisted Chemistry Problem Solving](https://mlsanity.com/paper/2608.26164/) (2026)
- [CQ4OE: A benchmark for assessing LLM-assisted ontology generation from competency questions](https://mlsanity.com/paper/2609.26029/) (2026)

## Abstract (arXiv)

Competency question (CQ) formulation is central to several ontology development and evaluation methodologies. Traditionally, the task of crafting these competency questions heavily relies on the effort of domain experts and knowledge engineers which is often time-consuming and labor-intensive. With the emergence of Large Language Models (LLMs), there arises the possibility to automate and enhance this process. Unlike other similar works which use existing ontologies or knowledge graphs as input to LLMs, we present a retrieval-augmented generation (RAG) approach that uses LLMs for the automatic generation of CQs given a set of scientific papers considered to be a domain knowledge base. We investigate its performance and specifically, we study the impact of different number of papers to the RAG and different temperature setting of the LLM. We conduct experiments using GPT-4 on two domain ontology engineering tasks and compare results against ground-truth CQs constructed by domain experts. Empirical assessments on the results, utilizing evaluation metrics (precision and consistency), reveal that compared to zero-shot prompting, adding relevant domain knowledge to the RAG improves the performance of LLMs on generating CQs for concrete ontology engineering tasks.