# OAEI-LLM: A Benchmark Dataset for Understanding Large Language Model Hallucinations in Ontology Matching

arXiv: https://arxiv.org/abs/2409.14038 · HTML: https://mlsanity.com/paper/2409.14038/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 15 references that are in the map.
- First posted on arXiv 21 Sep 2024.

## Builds on (most cited first, 7 of 7)

- [Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models](https://mlsanity.com/paper/2309.01219/) (2023)
- [LLMs4OL: Large Language Models for Ontology Learning](https://mlsanity.com/paper/2307.16648/) (2023)
- [OLaLa: Ontology Matching with Large Language Models](https://mlsanity.com/paper/2311.03837/) (2023)
- [Exploring Large Language Models for Ontology Alignment](https://mlsanity.com/paper/2309.07172/) (2023)
- [A Simple Standard for Sharing Ontological Mappings (SSSOM)](https://mlsanity.com/paper/2112.07051/) (2021)
- [Conversational Ontology Alignment with ChatGPT](https://mlsanity.com/paper/2308.09217/) (2023)
- [Towards Complex Ontology Alignment using Large Language Models](https://mlsanity.com/paper/2404.10329/) (2024)

## Built on by (most cited first, 3 of 3)

- [KROMA: Ontology Matching with Knowledge Retrieval and Large Language Models](https://mlsanity.com/paper/2507.14032/) (2025)
- [OAEI-LLM-T: A TBox Benchmark Dataset for Understanding Large Language Model Hallucinations in Ontology Matching](https://mlsanity.com/paper/2503.21813/) (2025)
- [PCFBench: A Diagnostic Benchmark for Product Carbon Footprint Estimation](https://mlsanity.com/paper/2608.27716/) (2026)

## Abstract (arXiv)

Hallucinations of large language models (LLMs) commonly occur in domain-specific downstream tasks, with no exception in ontology matching (OM). The prevalence of using LLMs for OM raises the need for benchmarks to better understand LLM hallucinations. The OAEI-LLM dataset is an extended version of the Ontology Alignment Evaluation Initiative (OAEI) datasets that evaluate LLM-specific hallucinations in OM tasks. We outline the methodology used in dataset construction and schema extension, and provide examples of potential use cases.