# Language Models as Ontology Encoders

arXiv: https://arxiv.org/abs/2507.14334 · HTML: https://mlsanity.com/paper/2507.14334/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 42 references that are in the map.
- First posted on arXiv 18 Jul 2025.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Poincar'e Embeddings for Learning Hierarchical Representations](https://mlsanity.com/paper/1705.08039/) (2017)
- [KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation](https://mlsanity.com/paper/1911.06136/) (2019)
- [KG-BERT: BERT for Knowledge Graph Completion](https://mlsanity.com/paper/1909.03193/) (2019)
- [Hyperbolic Entailment Cones for Learning Hierarchical Embeddings](https://mlsanity.com/paper/1804.01882/) (2018)
- [OWL2Vec*: Embedding of OWL Ontologies](https://mlsanity.com/paper/2009.14654/) (2020)
- [EL Embeddings: Geometric construction of models for the Description Logic EL ++](https://mlsanity.com/paper/1902.10499/) (2019)
- [BERTMap: A BERT-based Ontology Alignment System](https://mlsanity.com/paper/2112.02682/) (2021)
- [Contextual Semantic Embeddings for Ontology Subsumption Prediction](https://mlsanity.com/paper/2202.09791/) (2022)
- [Language Models as Hierarchy Encoders](https://mlsanity.com/paper/2401.11374/) (2024)
- [OPA2Vec: combining formal and informal content of biomedical ontologies to improve similarity-based prediction](https://mlsanity.com/paper/1804.10922/) (2018)
- [Faithiful Embeddings for EL++ Knowledge Bases](https://mlsanity.com/paper/2201.09919/) (2022)
- [Dual Box Embeddings for the Description Logic EL++](https://mlsanity.com/paper/2301.11118/) (2023)
- [DeepOnto: A Python Package for Ontology Engineering with Deep Learning](https://mlsanity.com/paper/2307.03067/) (2023)
- [Description Logic EL++ Embeddings with Intersectional Closure](https://mlsanity.com/paper/2202.14018/) (2022)
- [Ontology Embedding: A Survey of Methods, Applications and Resources](https://mlsanity.com/paper/2406.10964/) (2024)
- [Truveta Mapper: A Zero-shot Ontology Alignment Framework](https://mlsanity.com/paper/2301.09767/) (2023)
- [TransBox: EL++-closed Ontology Embedding](https://mlsanity.com/paper/2410.14571/) (2024)
- [Ontology Completion with Natural Language Inference and Concept Embeddings: An Analysis](https://mlsanity.com/paper/2403.17216/) (2024)
- [RegD: Hierarchical Embeddings via Dissimilarity between Arbitrary Euclidean Regions](https://mlsanity.com/paper/2501.17518/) (2025)

## Built on by (most cited first, 5 of 5)

- [Hierarchical Retrieval with Out-Of-Vocabulary Queries: A Case Study on SNOMED CT](https://mlsanity.com/paper/2511.16698/) (2025)
- [NeurOWL: An LLM-Based Neural-symbolic Framework for Incomplete OWL Ontology Reasoning](https://mlsanity.com/paper/2607.15776/) (2026)
- [AgentMap: Joint Equivalence and Subsumption Discovery for Ontology Matching](https://mlsanity.com/paper/2607.27130/) (2026)
- [HCOE: Hyperbolic Clinical Ontology Embeddings from Biomedical Language Models](https://mlsanity.com/paper/2609.30763/) (2026)
- [SMI: Ontology-Grounded Semantic IDs for Hierarchy-Aware Medical Concept Representation](https://mlsanity.com/paper/2610.18672/) (2026)

## Abstract (arXiv)

OWL (Web Ontology Language) ontologies which are able to formally represent complex knowledge and support semantic reasoning have been widely adopted across various domains such as healthcare and bioinformatics. Recently, ontology embeddings have gained wide attention due to its potential to infer plausible new knowledge and approximate complex reasoning. However, existing methods face notable limitations: geometric model-based embeddings typically overlook valuable textual information, resulting in suboptimal performance, while the approaches that incorporate text, which are often based on language models, fail to preserve the logical structure. In this work, we propose a new ontology embedding method OnT, which tunes a Pretrained Language Model (PLM) via geometric modeling in a hyperbolic space for effectively incorporating textual labels and simultaneously preserving class hierarchies and other logical relationships of Description Logic EL. Extensive experiments on four real-world ontologies show that OnT consistently outperforms the baselines including the state-of-the-art across both tasks of prediction and inference of axioms. OnT also demonstrates strong potential in real-world applications, indicated by its robust transfer learning abilities and effectiveness in real cases of constructing a new ontology from SNOMED CT. Data and code are available at https://github.com/HuiYang1997/OnT.