# Ontology-Based Contextual AI Evaluations (OB-CAIE) Methodology

arXiv: https://arxiv.org/abs/2610.00529 · HTML: https://mlsanity.com/paper/2610.00529/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 4 of its 10 references that are in the map.
- First posted on arXiv 30 Sep 2026.

## Builds on (most cited first, 4 of 4)

- [Lessons from the Trenches on Reproducible Evaluation of Language Models](https://mlsanity.com/paper/2405.14782/) (2024)
- [Participation is not a Design Fix for Machine Learning](https://mlsanity.com/paper/2007.02423/) (2020)
- [Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarks](https://mlsanity.com/paper/2504.18838/) (2025)
- [Eval Factsheets: A Structured Framework for Documenting AI Evaluations](https://mlsanity.com/paper/2512.04062/) (2025)

## Abstract (arXiv)

The ontology-based contextual AI evaluation (OB-CAIE) methodology was developed to address a lack of scientific rigor that arises from unclear testing coverage, to balance human expertise and automations, and to address a lack of reproducibility of AI evaluation testing environments. OB-CAIE strengthens the current state of AI evaluations by addressing the first step in the scientific method by clearly defining what will be tested. Two ontologies represent the tractable problem space in the OB-CAIE methodology: the Domain-Specific Ontology (DSO) and the Evaluation Process Ontology (EPO). The DSO is the what; the EPO is the how. An OB-CAIE problem space can be used for one or multiple AI evaluations. The OB-CAIE methodology allows for human judgment at specific points, in scientifically grounded ways, and in complex subject areas where human feedback is genuinely irreducible or machine irreplaceable. A key advantage of the OB-CAIE methodology is that failure points can be traced, visualized and analyzed within the canonical OB-CAIE methodology problem space.