# Contextual Semantic Embeddings for Ontology Subsumption Prediction

arXiv: https://arxiv.org/abs/2202.09791 · HTML: https://mlsanity.com/paper/2202.09791/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 57 references that are in the map.
- First posted on arXiv 20 Feb 2022.

## Builds on (most cited first, 21 of 21)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [Embedding Entities and Relations for Learning and Inference in Knowledge Bases](https://mlsanity.com/paper/1412.6575/) (2014)
- [Poincar'e Embeddings for Learning Hierarchical Representations](https://mlsanity.com/paper/1705.08039/) (2017)
- [COMET: Commonsense Transformers for Automatic Knowledge Graph Construction](https://mlsanity.com/paper/1906.05317/) (2019)
- [KG-BERT: BERT for Knowledge Graph Completion](https://mlsanity.com/paper/1909.03193/) (2019)
- [Learning Hierarchy-Aware Knowledge Graph Embeddings for Link Prediction](https://mlsanity.com/paper/1911.09419/) (2019)
- [Knowledge Graph Embedding for Link Prediction: A Comparative Analysis](https://mlsanity.com/paper/2002.00819/) (2020)
- [Probabilistic Embedding of Knowledge Graphs with Box Lattice Measures](https://mlsanity.com/paper/1805.06627/) (2018)
- [OWL2Vec*: Embedding of OWL Ontologies](https://mlsanity.com/paper/2009.14654/) (2020)
- [EL Embeddings: Geometric construction of models for the Description Logic EL ++](https://mlsanity.com/paper/1902.10499/) (2019)
- [BERTMap: A BERT-based Ontology Alignment System](https://mlsanity.com/paper/2112.02682/) (2021)
- [A Survey on Knowledge Graph Embeddings with Literals: Which model links better Literal-ly?](https://mlsanity.com/paper/1910.12507/) (2019)
- [OPA2Vec: combining formal and informal content of biomedical ontologies to improve similarity-based prediction](https://mlsanity.com/paper/1804.10922/) (2018)
- [Onto2Vec: joint vector-based representation of biological entities and their ontology-based annotations](https://mlsanity.com/paper/1802.00864/) (2018)

## Built on by (most cited first, 30 of 31)

- [Large Language Models and Knowledge Graphs: Opportunities and Challenges](https://mlsanity.com/paper/2308.06374/) (2023)
- [Language Models as Hierarchy Encoders](https://mlsanity.com/paper/2401.11374/) (2024)
- [Language Model Analysis for Ontology Subsumption Inference](https://mlsanity.com/paper/2302.06761/) (2023)
- [Towards Ontology Construction with Language Models](https://mlsanity.com/paper/2309.09898/) (2023)
- [Machine Learning-Friendly Biomedical Datasets for Equivalence and Subsumption Ontology Matching](https://mlsanity.com/paper/2205.03447/) (2022)
- [Dual Box Embeddings for the Description Logic EL++](https://mlsanity.com/paper/2301.11118/) (2023)
- [DeepOnto: A Python Package for Ontology Engineering with Deep Learning](https://mlsanity.com/paper/2307.03067/) (2023)
- [Research Trends for the Interplay between Large Language Models and Knowledge Graphs](https://mlsanity.com/paper/2406.08223/) (2024)
- [Ontology Embedding: A Survey of Methods, Applications and Resources](https://mlsanity.com/paper/2406.10964/) (2024)
- [Knowledge Graphs for the Life Sciences: Recent Developments, Challenges and Opportunities](https://mlsanity.com/paper/2309.17255/) (2023)
- [Distilling Semantic Concept Embeddings from Contrastively Fine-Tuned Language Models](https://mlsanity.com/paper/2305.09785/) (2023)
- [Ontology Enrichment from Texts: A Biomedical Dataset for Concept Discovery and Placement](https://mlsanity.com/paper/2306.14704/) (2023)
- [Evaluating Knowledge Graph Based Retrieval Augmented Generation Methods under Knowledge Incompleteness](https://mlsanity.com/paper/2504.05163/) (2025)
- [TransBox: EL++-closed Ontology Embedding](https://mlsanity.com/paper/2410.14571/) (2024)
- [Language Models as Ontology Encoders](https://mlsanity.com/paper/2507.14334/) (2025)
- [Ontology Completion with Natural Language Inference and Concept Embeddings: An Analysis](https://mlsanity.com/paper/2403.17216/) (2024)
- [From axioms over graphs to vectors, and back again: evaluating the properties of graph-based ontology embeddings](https://mlsanity.com/paper/2303.16519/) (2023)
- [Prompting Disentangled Embeddings for Knowledge Graph Completion with Pre-trained Language Model](https://mlsanity.com/paper/2312.01837/) (2023)
- [A Language Model based Framework for New Concept Placement in Ontologies](https://mlsanity.com/paper/2402.17897/) (2024)
- [Using Large Language Models for OntoClean-based Ontology Refinement](https://mlsanity.com/paper/2403.15864/) (2024)
- [A Self-matching Training Method with Annotation Embedding Models for Ontology Subsumption Prediction](https://mlsanity.com/paper/2402.16278/) (2024)
- [Taxonomy Inference for Tabular Data Using Large Language Models](https://mlsanity.com/paper/2503.21810/) (2025)
- [GenOM: Ontology Matching with Description Generation and Large Language Model](https://mlsanity.com/paper/2508.10703/) (2025)
- [Current and Future Challenges in Knowledge Representation and Reasoning](https://mlsanity.com/paper/2308.04161/) (2023)
- [RelBERT: Embedding Relations with Language Models](https://mlsanity.com/paper/2310.00299/) (2023)
- [OWL2Vec4OA: Tailoring Knowledge Graph Embeddings for Ontology Alignment](https://mlsanity.com/paper/2408.06310/) (2024)
- [OntoAligner Meets Knowledge Graph Embedding Aligners](https://mlsanity.com/paper/2509.26417/) (2025)
- [Lattice-preserving ALC ontology embeddings with saturation](https://mlsanity.com/paper/2305.07163/) (2023)
- [Can Structured Data Reduce Epistemic Uncertainty?](https://mlsanity.com/paper/2410.11141/) (2024)
- [NeurOWL: An LLM-Based Neural-symbolic Framework for Incomplete OWL Ontology Reasoning](https://mlsanity.com/paper/2607.15776/) (2026)

## Abstract (arXiv)

Automating ontology construction and curation is an important but challenging task in knowledge engineering and artificial intelligence. Prediction by machine learning techniques such as contextual semantic embedding is a promising direction, but the relevant research is still preliminary especially for expressive ontologies in Web Ontology Language (OWL). In this paper, we present a new subsumption prediction method named BERTSubs for classes of OWL ontology. It exploits the pre-trained language model BERT to compute contextual embeddings of a class, where customized templates are proposed to incorporate the class context (e.g., neighbouring classes) and the logical existential restriction. BERTSubs is able to predict multiple kinds of subsumers including named classes from the same ontology or another ontology, and existential restrictions from the same ontology. Extensive evaluation on five real-world ontologies for three different subsumption tasks has shown the effectiveness of the templates and that BERTSubs can dramatically outperform the baselines that use (literal-aware) knowledge graph embeddings, non-contextual word embeddings and the state-of-the-art OWL ontology embeddings.