# Ontology Learning with LLMs: A Benchmark Study on Axiom Identification

arXiv: https://arxiv.org/abs/2512.05594 · HTML: https://mlsanity.com/paper/2512.05594/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 51 references that are in the map.
- First posted on arXiv 5 Dec 2025.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [The Flan Collection: Designing Data and Methods for Effective Instruction Tuning](https://mlsanity.com/paper/2301.13688/) (2023)
- [MEDEC: A Benchmark for Medical Error Detection and Correction in Clinical Notes](https://mlsanity.com/paper/2412.19260/) (2024)
- [LLMs4OL: Large Language Models for Ontology Learning](https://mlsanity.com/paper/2307.16648/) (2023)
- [GPT-RE: In-context Learning for Relation Extraction using Large Language Models](https://mlsanity.com/paper/2305.02105/) (2023)
- [Ontology Generation using Large Language Models](https://mlsanity.com/paper/2503.05388/) (2025)
- [Ontology engineering with Large Language Models](https://mlsanity.com/paper/2307.16699/) (2023)
- [The representation landscape of few-shot learning and fine-tuning in large language models](https://mlsanity.com/paper/2409.03662/) (2024)
- [End-to-End Ontology Learning with Large Language Models](https://mlsanity.com/paper/2410.23584/) (2024)
- [LLMs4Life: Large Language Models for Ontology Learning in Life Sciences](https://mlsanity.com/paper/2412.02035/) (2024)
- [A Looming Replication Crisis in Evaluating Behavior in Language Models? Evidence and Solutions](https://mlsanity.com/paper/2409.20303/) (2024)
- [Large language models as oracles for instantiating ontologies with domain-specific knowledge](https://mlsanity.com/paper/2404.04108/) (2024)
- [Predicting the Score of Atomic Candidate OWL Class Axioms](https://mlsanity.com/paper/2212.10841/) (2022)

## Built on by (most cited first, 4 of 4)

- [Structured Decomposition for LLM Reasoning: Cross-Domain Validation and Semantic Web Integration](https://mlsanity.com/paper/2601.01609/) (2026)
- [Provably Auditable and Safe LLM Agents from Human-Authored Ontologies](https://mlsanity.com/paper/2606.04903/) (2026)
- [pro-team at LLMs4OL 2026 Tasks Flagship and Reuse: Retrieval-Augmented Generation and Vocabulary-Constrained Filtering for Ontology Learning](https://mlsanity.com/paper/2608.27101/) (2026)
- [CQ4OE: A benchmark for assessing LLM-assisted ontology generation from competency questions](https://mlsanity.com/paper/2609.26029/) (2026)

## Abstract (arXiv)

Ontologies are an important tool for structuring domain knowledge, but their development is a complex task that requires significant modelling and domain expertise. Ontology learning, aimed at automating this process, has seen advancements in the past decade with the improvement of Natural Language Processing techniques, and especially with the recent growth of Large Language Models (LLMs). This paper investigates the challenge of identifying axioms: fundamental ontology components that define logical relations between classes and properties. In this work, we introduce an Ontology Axiom Benchmark OntoAxiom, and systematically test LLMs on that benchmark for axiom identification, evaluating different prompting strategies, ontologies, and axiom types. The benchmark consists of nine medium-sized ontologies with together 17.118 triples, and 2.771 axioms. We focus on subclass, disjoint, subproperty, domain, and range axioms. To evaluate LLM performance, we compare twelve LLMs with three shot settings and two prompting strategies: a Direct approach where we query all axioms at once, versus an Axiom-by-Axiom (AbA) approach, where each prompt queries for one axiom only. Our findings show that the AbA prompting leads to higher F1 scores than the direct approach. However, performance varies across axioms, suggesting that certain axioms are more challenging to identify. The domain also influences performance: the FOAF ontology achieves a score of 0.642 for the subclass axiom, while the music ontology reaches only 0.218. Larger LLMs outperform smaller ones, but smaller models may still be viable for resource-constrained settings. Although performance overall is not high enough to fully automate axiom identification, LLMs can provide valuable candidate axioms to support ontology engineers with the development and refinement of ontologies.