# OAEI-LLM-T: A TBox Benchmark Dataset for Understanding Large Language Model Hallucinations in Ontology Matching

arXiv: https://arxiv.org/abs/2503.21813 · HTML: https://mlsanity.com/paper/2503.21813/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 41 references that are in the map.
- First posted on arXiv 25 Mar 2025.

## Builds on (most cited first, 30 of 31)

- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://mlsanity.com/paper/2311.05232/) (2023)
- [Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models](https://mlsanity.com/paper/2309.01219/) (2023)
- [ORPO: Monolithic Preference Optimization without Reference Model](https://mlsanity.com/paper/2403.07691/) (2024)
- [HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models](https://mlsanity.com/paper/2305.11747/) (2023)
- [Hallucination is Inevitable: An Innate Limitation of Large Language Models](https://mlsanity.com/paper/2401.11817/) (2024)
- [FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation](https://mlsanity.com/paper/2310.03214/) (2023)
- [RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models](https://mlsanity.com/paper/2401.00396/) (2023)
- [ToolQA: A Dataset for LLM Question Answering with External Tools](https://mlsanity.com/paper/2306.13304/) (2023)
- [RealTime QA: What's the Answer Right Now?](https://mlsanity.com/paper/2207.13332/) (2022)
- [LLM Lies: Hallucinations are not Bugs, but Features as Adversarial Examples](https://mlsanity.com/paper/2310.01469/) (2023)
- [Med-HALT: Medical Domain Hallucination Test for Large Language Models](https://mlsanity.com/paper/2307.15343/) (2023)
- [Generating Benchmarks for Factuality Evaluation of Language Models](https://mlsanity.com/paper/2307.06908/) (2023)
- [The Dawn After the Dark: An Empirical Study on Factuality Hallucination in Large Language Models](https://mlsanity.com/paper/2401.03205/) (2024)
- [KoLA: Carefully Benchmarking World Knowledge of Large Language Models](https://mlsanity.com/paper/2306.09296/) (2023)
- [TofuEval: Evaluating Hallucinations of LLMs on Topic-Focused Dialogue Summarization](https://mlsanity.com/paper/2402.13249/) (2024)
- [DelucionQA: Detecting Hallucinations in Domain-specific Question Answering](https://mlsanity.com/paper/2312.05200/) (2023)
- [Confabulation: The Surprising Value of Large Language Model Hallucinations](https://mlsanity.com/paper/2406.04175/) (2024)
- [OLaLa: Ontology Matching with Large Language Models](https://mlsanity.com/paper/2311.03837/) (2023)
- [Evaluating Hallucinations in Chinese Large Language Models](https://mlsanity.com/paper/2310.03368/) (2023)
- [Agent-OM: Leveraging LLM Agents for Ontology Matching](https://mlsanity.com/paper/2312.00326/) (2023)
- [HaluEval-Wild: Evaluating Hallucinations of Language Models in the Wild](https://mlsanity.com/paper/2403.04307/) (2024)
- [Exploring Large Language Models for Ontology Alignment](https://mlsanity.com/paper/2309.07172/) (2023)
- [LLMs4OM: Matching Ontologies with Large Language Models](https://mlsanity.com/paper/2404.10317/) (2024)
- [MedHallBench: A New Benchmark for Assessing Hallucination in Medical Large Language Models](https://mlsanity.com/paper/2412.18947/) (2024)
- [Conversational Ontology Alignment with ChatGPT](https://mlsanity.com/paper/2308.09217/) (2023)
- [Towards Complex Ontology Alignment using Large Language Models](https://mlsanity.com/paper/2404.10329/) (2024)
- [DAHL: Domain-specific Automated Hallucination Evaluation of Long-Form Text through a Benchmark Dataset in Biomedicine](https://mlsanity.com/paper/2411.09255/) (2024)

## Built on by (most cited first, 1 of 1)

- [PCFBench: A Diagnostic Benchmark for Product Carbon Footprint Estimation](https://mlsanity.com/paper/2608.27716/) (2026)

## Abstract (arXiv)

Hallucinations are often inevitable in downstream tasks using large language models (LLMs). To tackle the substantial challenge of addressing hallucinations for LLM-based ontology matching (OM) systems, we introduce a new benchmark dataset OAEI-LLM-T. The dataset evolves from seven TBox datasets in the Ontology Alignment Evaluation Initiative (OAEI), capturing hallucinations of ten different LLMs performing OM tasks. These OM-specific hallucinations are organised into two primary categories and six sub-categories. We showcase the usefulness of the dataset in constructing an LLM leaderboard for OM tasks and for fine-tuning LLMs used in OM tasks.