# Self-attention Presents Low-dimensional Knowledge Graph Embeddings for Link Prediction

arXiv: https://arxiv.org/abs/2112.10644 · HTML: https://mlsanity.com/paper/2112.10644/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 42 references that are in the map.
- First posted on arXiv 20 Dec 2021.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Modeling Relational Data with Graph Convolutional Networks](https://mlsanity.com/paper/1703.06103/) (2017)
- [Embedding Entities and Relations for Learning and Inference in Knowledge Bases](https://mlsanity.com/paper/1412.6575/) (2014)
- [Complex Embeddings for Simple Link Prediction](https://mlsanity.com/paper/1606.06357/) (2016)
- [RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space](https://mlsanity.com/paper/1902.10197/) (2019)
- [Convolutional 2D Knowledge Graph Embeddings](https://mlsanity.com/paper/1707.01476/) (2017)
- [Are Transformers universal approximators of sequence-to-sequence functions?](https://mlsanity.com/paper/1912.10077/) (2019)
- [Composition-based Multi-Relational Graph Convolutional Networks](https://mlsanity.com/paper/1911.03082/) (2019)
- [KG-BERT: BERT for Knowledge Graph Completion](https://mlsanity.com/paper/1909.03193/) (2019)
- [TuckER: Tensor Factorization for Knowledge Graph Completion](https://mlsanity.com/paper/1901.09590/) (2019)
- [A System for Massively Parallel Hyperparameter Tuning](https://mlsanity.com/paper/1810.05934/) (2018)
- [SimplE Embedding for Link Prediction in Knowledge Graphs](https://mlsanity.com/paper/1802.04868/) (2018)
- [Low-Dimensional Hyperbolic Knowledge Graph Embeddings](https://mlsanity.com/paper/2005.00545/) (2020)
- [Holographic Embeddings of Knowledge Graphs](https://mlsanity.com/paper/1510.04935/) (2015)
- [Quaternion Knowledge Graph Embeddings](https://mlsanity.com/paper/1904.10281/) (2019)
- [DRUM: End-To-End Differentiable Rule Mining On Knowledge Graphs](https://mlsanity.com/paper/1911.00055/) (2019)
- [Go for a Walk and Arrive at the Answer: Reasoning Over Paths in Knowledge Bases using Reinforcement Learning](https://mlsanity.com/paper/1711.05851/) (2017)
- [Canonical Tensor Decomposition for Knowledge Base Completion](https://mlsanity.com/paper/1806.07297/) (2018)
- [Multi-relational Poincar'e Graph Embeddings](https://mlsanity.com/paper/1905.09791/) (2019)
- [A Re-evaluation of Knowledge Graph Completion Methods](https://mlsanity.com/paper/1911.03903/) (2019)
- [InteractE: Improving Convolution-based Knowledge Graph Embeddings by Increasing Feature Interactions](https://mlsanity.com/paper/1911.00219/) (2019)
- [HittER: Hierarchical Transformers for Knowledge Graph Embeddings](https://mlsanity.com/paper/2008.12813/) (2020)
- [On the Expressive Power of Self-Attention Matrices](https://mlsanity.com/paper/2106.03764/) (2021)
- [Hypernetwork Knowledge Graph Embeddings](https://mlsanity.com/paper/1808.07018/) (2018)
- [CoKE: Contextualized Knowledge Graph Embedding](https://mlsanity.com/paper/1911.02168/) (2019)
- [Mutual Information Scaling and Expressive Power of Sequence Models](https://mlsanity.com/paper/1905.04271/) (2019)

## Built on by (most cited first, 8 of 8)

- [HyCubE: Efficient Knowledge Hypergraph 3D Circular Convolutional Embedding](https://mlsanity.com/paper/2402.08961/) (2024)
- [Fully Hyperbolic Rotation for Knowledge Graph Embedding](https://mlsanity.com/paper/2411.03622/) (2024)
- [Efficient Relation-aware Neighborhood Aggregation in Graph Neural Networks via Tensor Decomposition](https://mlsanity.com/paper/2212.05581/) (2022)
- [ConvD: Attention Enhanced Dynamic Convolutional Embeddings for Knowledge Graph Completion](https://mlsanity.com/paper/2312.07589/) (2023)
- [DuetGraph: Coarse-to-Fine Knowledge Graph Reasoning with Dual-Pathway Global-Local Fusion](https://mlsanity.com/paper/2507.11229/) (2025)
- [A Bird's-Eye Tutorial of Graph Attention Architectures](https://mlsanity.com/paper/2206.02849/) (2022)
- [Toward Fair Graph Neural Networks Via Dual-Teacher Knowledge Distillation](https://mlsanity.com/paper/2412.00382/) (2024)
- [A set of semantic data flow diagrams and its security analysis based on ontologies and knowledge graphs](https://mlsanity.com/paper/2303.11198/) (2023)

## Abstract (arXiv)

A few models have tried to tackle the link prediction problem, also known as knowledge graph completion, by embedding knowledge graphs in comparably lower dimensions. However, the state-of-the-art results are attained at the cost of considerably increasing the dimensionality of embeddings which causes scalability issues in the case of huge knowledge bases. Transformers have been successfully used recently as powerful encoders for knowledge graphs, but available models still have scalability issues. To address this limitation, we introduce a Transformer-based model to gain expressive low-dimensional embeddings. We utilize a large number of self-attention heads as the key to applying query-dependent projections to capture mutual information between entities and relations. Empirical results on WN18RR and FB15k-237 as standard link prediction benchmarks demonstrate that our model has favorably comparable performance with the current state-of-the-art models. Notably, we yield our promising results with a significant reduction of 66.9% in the dimensionality of embeddings compared to the five best recent state-of-the-art competitors on average.