# Language Models as Knowledge Embeddings

arXiv: https://arxiv.org/abs/2206.12617 · HTML: https://mlsanity.com/paper/2206.12617/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 30 references that are in the map.
- First posted on arXiv 25 Jun 2022.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Embedding Entities and Relations for Learning and Inference in Knowledge Bases](https://mlsanity.com/paper/1412.6575/) (2014)
- [Complex Embeddings for Simple Link Prediction](https://mlsanity.com/paper/1606.06357/) (2016)
- [RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space](https://mlsanity.com/paper/1902.10197/) (2019)
- [Convolutional 2D Knowledge Graph Embeddings](https://mlsanity.com/paper/1707.01476/) (2017)
- [KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation](https://mlsanity.com/paper/1911.06136/) (2019)
- [KG-BERT: BERT for Knowledge Graph Completion](https://mlsanity.com/paper/1909.03193/) (2019)
- [TuckER: Tensor Factorization for Knowledge Graph Completion](https://mlsanity.com/paper/1901.09590/) (2019)
- [Learning Hierarchy-Aware Knowledge Graph Embeddings for Link Prediction](https://mlsanity.com/paper/1911.09419/) (2019)
- [CoKE: Contextualized Knowledge Graph Embedding](https://mlsanity.com/paper/1911.02168/) (2019)
- [Triple Classification for Scholarly Knowledge Graph Completion](https://mlsanity.com/paper/2111.11845/) (2021)

## Built on by (most cited first, 14 of 14)

- [Unifying Large Language Models and Knowledge Graphs: A Roadmap](https://mlsanity.com/paper/2306.08302/) (2023)
- [LLMs for Knowledge Graph Construction and Reasoning: Recent Capabilities and Future Opportunities](https://mlsanity.com/paper/2305.13168/) (2023)
- [Editing Language Model-based Knowledge Graph Embeddings](https://mlsanity.com/paper/2301.10405/) (2023)
- [KG-FIT: Knowledge Graph Fine-Tuning Upon Open-World Knowledge](https://mlsanity.com/paper/2405.16412/) (2024)
- [Knowledge Graph Embedding: An Overview](https://mlsanity.com/paper/2309.12501/) (2023)
- [Incorporating Structured Sentences with Time-enhanced BERT for Fully-inductive Temporal Relation Prediction](https://mlsanity.com/paper/2304.04717/) (2023)
- [Large Language Model Enhanced Knowledge Representation Learning: A Survey](https://mlsanity.com/paper/2407.00936/) (2024)
- [Knowledge Graphs and Pre-trained Language Models enhanced Representation Learning for Conversational Recommender Systems](https://mlsanity.com/paper/2312.10967/) (2023)
- [MoCoSA: Momentum Contrast for Knowledge Graph Completion with Structure-Augmented Pre-trained Language Models](https://mlsanity.com/paper/2308.08204/) (2023)
- [Language Models sounds the Death Knell of Knowledge Graphs](https://mlsanity.com/paper/2301.03980/) (2023)
- [Enhancing Heterogeneous Knowledge Graph Completion with a Novel GAT-based Approach](https://mlsanity.com/paper/2408.02456/) (2024)
- [KG-BiLM: Knowledge Graph Embedding via Bidirectional Language Models](https://mlsanity.com/paper/2506.03576/) (2025)
- [On the Theoretical Limitations of Embedding-based Link Prediction](https://mlsanity.com/paper/2506.22271/) (2025)
- [Progressive Distillation Based on Masked Generation Feature Method for Knowledge Graph Completion](https://mlsanity.com/paper/2401.12997/) (2024)

## Abstract (arXiv)

Knowledge embeddings (KE) represent a knowledge graph (KG) by embedding entities and relations into continuous vector spaces. Existing methods are mainly structure-based or description-based. Structure-based methods learn representations that preserve the inherent structure of KGs. They cannot well represent abundant long-tail entities in real-world KGs with limited structural information. Description-based methods leverage textual information and language models. Prior approaches in this direction barely outperform structure-based ones, and suffer from problems like expensive negative sampling and restrictive description demand. In this paper, we propose LMKE, which adopts Language Models to derive Knowledge Embeddings, aiming at both enriching representations of long-tail entities and solving problems of prior description-based methods. We formulate description-based KE learning with a contrastive learning framework to improve efficiency in training and evaluation. Experimental results show that LMKE achieves state-of-the-art performance on KE benchmarks of link prediction and triple classification, especially for long-tail entities.