# A curated, ontology-based, large-scale knowledge graph of artificial intelligence tasks and benchmarks

arXiv: https://arxiv.org/abs/2110.01434 · HTML: https://mlsanity.com/paper/2110.01434/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 40 references that are in the map.
- First posted on arXiv 4 Oct 2021.

## Builds on (most cited first, 7 of 7)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [The AI Index 2021 Annual Report](https://mlsanity.com/paper/2103.06312/) (2021)
- [OpenBioLink: A benchmarking framework for large-scale biomedical link prediction](https://mlsanity.com/paper/1912.04616/) (2019)
- [A critical analysis of metrics used for measuring progress in artificial intelligence](https://mlsanity.com/paper/2008.02577/) (2020)
- [Nanopublications: A Growing Resource of Provenance-Centric Scientific Linked Data](https://mlsanity.com/paper/1809.06532/) (2018)
- [WebProt'eg'e: A Cloud-Based Ontology Editor](https://mlsanity.com/paper/1902.08251/) (2019)
- [Pattern-based design applied to cultural heritage knowledge graphs](https://mlsanity.com/paper/1911.07585/) (2019)

## Built on by (most cited first, 11 of 11)

- [Mapping global dynamics of benchmark creation and saturation in artificial intelligence](https://mlsanity.com/paper/2203.04592/) (2022)
- [A global analysis of metrics used for measuring performance in natural language processing](https://mlsanity.com/paper/2204.11574/) (2022)
- [Benchmark datasets driving artificial intelligence development fail to capture the needs of medical professionals](https://mlsanity.com/paper/2201.07040/) (2022)
- [A Review on Knowledge Graphs for Healthcare: Resources, Applications, and Promises](https://mlsanity.com/paper/2306.04802/) (2023)
- [SciDMT: A Large-Scale Corpus for Detecting Scientific Mentions](https://mlsanity.com/paper/2406.14756/) (2024)
- [The Artificial Intelligence Ontology: LLM-assisted construction of AI concept hierarchies](https://mlsanity.com/paper/2404.03044/) (2024)
- [LightLLM: A Versatile Large Language Model for Predictive Light Sensing](https://mlsanity.com/paper/2411.15211/) (2024)
- [AI Thinking: A framework for rethinking artificial intelligence in practice](https://mlsanity.com/paper/2409.12922/) (2024)
- [Ontology of Belief Diversity: A Community-Based Epistemological Approach](https://mlsanity.com/paper/2408.01455/) (2024)
- [Ontology-supported AI Model and Dataset Management](https://mlsanity.com/paper/2608.21224/) (2026)
- [SAREF-based Ontology for Distributed AI Workflows across the Edge-Fog-Cloud Continuum](https://mlsanity.com/paper/2608.26160/) (2026)

## Abstract (arXiv)

Research in artificial intelligence (AI) is addressing a growing number of tasks through a rapidly growing number of models and methodologies. This makes it difficult to keep track of where novel AI methods are successfully -- or still unsuccessfully -- applied, how progress is measured, how different advances might synergize with each other, and how future research should be prioritized. To help address these issues, we created the Intelligence Task Ontology and Knowledge Graph (ITO), a comprehensive, richly structured and manually curated resource on artificial intelligence tasks, benchmark results and performance metrics. The current version of ITO contain 685,560 edges, 1,100 classes representing AI processes and 1,995 properties representing performance metrics. The goal of ITO is to enable precise and network-based analyses of the global landscape of AI tasks and capabilities. ITO is based on technologies that allow for easy integration and enrichment with external data, automated inference and continuous, collaborative expert curation of underlying ontological models. We make the ITO dataset and a collection of Jupyter notebooks utilising ITO openly available.